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The  Government  Performance  and  Results  Act  of  1993  requires  government  executives  to 
focus  on  defining  missions,  setting  goals,  measuring  performance,  and  reporting 
accomplishments.  In  addition,  with  the  passage  of  the  Federal  Acquisition  Streamlining 
Act  of  1994  (FASA)  and  the  Clinger-Cohen  Act  of  1996,  performance-based  and  results- 
oriented  decision-making  is  now  required  for  all  major  investments  in  information 
technology  (IT).  Clearly,  this  intense  focus  on  results  is  one  of  the  most  important 
management  issues  now  confronting  federal  agencies. 

To  assist  federal  agencies  in  understanding  and  devising  effective  IT  measurement 
implementation  approaches,  we  examined  certain  public  and  private  organizations  well- 
known  for  their  IT  performance  leadership  and  management  expertise.  Similar  to  our 
past  efforts  examining  comprehensive  information  management  practices  of  other  leading 
organizations,1  we  have  taken  the  lessons  learned  from  these  organizations  and  developed 
a  suggested  framework  for  agencies  to  consider  when  designing  and  implementing  their 
IT  performance  management  approaches.  We  have  briefed  numerous  Chief  Information 
Officers,  agency  executives,  and  agency  IT  managers  on  our  work  as  part  of  our  effort  to 
advance  a  pragmatic  understanding  of  what  is  required  to  effectively  measure  the 
contribution  of  IT  to  mission  performance  and  program  outcomes. 

By  using  comprehensive  performance  information,  more  informed  decisions  can  be  made 
about  IT  investments  at  a  time  when  resources  are  limited  and  public  demands  for  better 
government  service  are  high.  Ultimately,  the  success  of  results-oriented  reform  legislation 
will  demand  concerted  management  effort  and  long-term  commitment.  The  key  practices 
and  steps  outlined  in  this  guide  can  help  agencies  achieve  success. 

This  report  was  prepared  under  the  direction  of  Dave  McClure,  Senior  Assistant  Director 
for  Information  Resource  Management  Policies  and  Issues.  If  you  have  questions  or 
comments  about  the  report,  he  can  be  reached  at  (202)  512-6257.  Other  major 
contributors  are  listed  in  appendix  IV. 


Accounting  and  Information  Management  Division 


Executive  Guide:  Improving  Mission  Performance  Through  Strategic  Information  Management  and 
Technology-Learning  From  Leading  Organizations  (GAO/AIMD-94-115),  May  1994;  Assessing  Risks  anri 
Returns:  A  Guide  for  Evaluating  Federal  Agencies'  IT  Investment,  Decision-making  (GAO/AIMD-IO.1.13), 
February  1997. 
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The  Demand  for  Performance  Management 


Increasingly,  federal  policymakers  are  insisting  that  government 
executives  provide  hard  facts  on  mission  and  program  results.  Program 
authorizations,  resource  decisions,  and  oversight  requirements 
increasingly  hinge  on  how  well  agencies  perform  against  expectations 
and  improve  performance  over  time.  As  such,  a  new  standard  for 
management  expertise  is  evolving:  setting  performance  targets,  designing 
efficiency  and  effectiveness  measures,  systematically  and  accurately 
measuring  outcomes,  and  then  using  the  results  for  informed  decision¬ 
making. 

Information  technology  (IT)  products,  services,  and  delivery  processes 
are  important  resources  for  results-driven  government  programs  and 
operations.  For  purposes  of  this  guide,  IT  also  includes  the 
organizational  units  and  contractors  primarily  responsible  for  delivering 
IT.  Line  managers— the  operational  customers2  relying  on  IT  products 
and  services-and  IT  managers  themselves,  want  to  know  "How  are 
information  technology  products  and  services,  including  the  information 
infrastructure,  supporting  the  delivery  and  effectiveness  of  the 
enterprise's  (agency)  programs?"  As  we  pointed  out  in  an  earlier  report, 
successful  organizations  rely  heavily  on  performance  measures  to 
operationalize  mission  goals  and  objectives,  quantify  problems,  evaluate 
alternatives,  allocate  resources,  track  progress,  and  learn  from  mistakes.3 
Operational  customers  and  IT  managers  in  these  organizations  form 
partnerships  to  design,  manage,  and  evaluate  IT  systems  that  are  critical 
to  achieving  improved  mission  success. 

Performance  Management: 

What  Are  the  Benefits? 

In  an  effective  performance  management  approach,  measures  are  not 
used  for  assigning  blame  or  to  unknowingly  comply  with  reporting 
requirements.  Quite  simply,  they  are  used  to  create  and  facilitate  action 
to  improve  performance.  Measures  and  performance  information  must 
link  to  strategic  management  processes.  An  effective  performance 


2For  purposes  of  this  guide,  an  "operational  customer"  is  the  end  user  of  the  IT 
products  and  services  (i.e.,  the  business  function  supported  by  the  IT  product  or 
service).  In  some  instances,  operational  customers  can  include  external  organizations 
and  entities  who  also  directly  use  that  agency's  IT  products  and  services  to  assist  in 
mission  support. 

3GAO/AIMD-94-l  15. 
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management  system  produces  information  that  delivers  the  following 
benefits.4 

•  Provides  an  early  warning  indicator  of  problems,  and  the 
effectiveness  of  corrective  action. 

•  Provides  input  to  resource  allocation  and  planning.  It  can  help 
organizations  prepare  for  future  conditions  that  likely  will  impact 
program  and  support  function  operations  and  the  demands  for 
products  and  services,  such  as  decreasing  personnel  or  financial 
resources  or  changes  in  workload.  Use  of  measures  can  give 
organizations  lead  times  for  needed  resource  adjustments  if  these 
conditions  are  known  in  advance. 

•  Provides  periodic  feedback  to  employees,  customers,  stakeholders, 
and  the  general  public  about  the  quality,  quantity,  cost,  and  timeliness 
of  products  and  services. 

Perhaps  most  importantly,  measures  build  a  common  results  language 
among  all  decision-makers.  Selected  measures  define  what  is  important 
to  an  organization,  what  it  holds  itself  accountable  for,  how  it  defines 
success,  and  how  it  structures  its  improvement  efforts. 


Results-Oriented  Legislation  Provides 
Impetus  for  Performance  Based  Management 

For  the  past  several  years,  the  Congress  has  emphasized  federal 
performance  improvement,  accountability  for  achieving  results,  and  cost 
reduction.  The  Chief  Financial  Officers  Act  (CFO)  of  1990,  the 
Government  Performance  and  Results  Act  (Results  Act)  of  1993,  the 
Federal  Acquisition  Streamlining  Act  (FASA)  of  1994,  the  Paperwork 
Reduction  Act  of  1995,  and  the  Clinger-Cohen  Act  of  1996  require 
agencies  to  demonstrate  improvements  in  IT  performance  management. 
These  laws  reinforce  financial  accountability,  emphasize  results-oriented 
management,  define  cost  performance  and  schedule  goals,  and  improve 
the  acquisition  of  IT  to  streamline  federal  programs. 

Under  the  CFO  Act,  CFOs  are  responsible  for  developing  and  maintaining 
integrated  accounting  and  financial  management  systems  that  include 
systematic  measurement  information  on  agency  performance.  OMB's 


“Jack  A  Brizius  and  Michael  D.  Campbell,  Getting  Results  (Washington.  D.C.,  Council  of 
Governors'  Policy  Advisors,  1991)  and  National  Academy  of  Public  Administration,  A 
Guide  to  Designing  Performance  Indicators  (Washington,  D.C.:  National  Academy  of 
Public  Administration,  November  1993). 
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Management  Circular  A- 11  (agency  guidance  for  preparing  and  submitting 
budget  estimates)  encourages  agencies  to  review  program  performance 
information  contained  in  the  most  recent  financial  statements  prepared 
under  the  CFO  Act  when  developing  their  program  performance 
indicators.  The  Results  Act  directs  federal  agencies  to  improve  their 
program  management  by  implementing  outcome-oriented  performance 
measurement  systems.  Agencies  are  to  prepare  annual  performance 
plans  with  objective,  quantifiable,  and  measurable  performance  indicators 
to  measure  relevant  outputs,  service  levels,  and  outcomes  of  each 
program  activity.  In  their  plans,  agencies  are  to  discuss  the  resources 
they  plan  to  use  to  achieve  their  intended  performance.  Information 
technology  is  a  critical  resource,  and  to  be  most  useful,  plans  should 
discuss  how  information  technology  will  be  applied  to  improve 
performance.5  Meeting  these  requirements  is  critical  in  developing  the 
mission  goals  and  performance  expectations  which  IT  products  and 
services  will  support.  Our  Executive  Guide:  Effectively  Implementing  the 
Government  Performance  and  Results  Act  contains  additional  guidance 
on  results-oriented  management  under  the  Results  Act.6 

The  Federal  Acquisition  Streamlining  Act  requires  federal  agencies  to 
assess  major  acquisition  cost,  performance,  and  scheduling.  Agency 
heads  must  determine  if  there  is  a  continuing  need  for  programs  that  are 
significantly  behind  schedule,  over  budget,  or  not  in  compliance  with 
performance  or  capability  requirements.  Congressional  policy  is  that 
each  executive  agency  should  achieve,  on  average,  90  percent  of  the  cost 
and  schedule  goals  established  for  agency  programs. 

The  Paperwork  Reduction  Act  requires  agencies  to  establish  information 
resources  management  goals  for  improving  the  productivity,  efficiency, 
and  effectiveness  of  agency  operations  and  methods  for  measuring 
progress  in  achieving  the  goals.  OMB's  Circular  A-130  (guidance 
resulting  from  the  Paperwork  Reduction  Act)  highlights  the  importance 
of  evaluation  and  performance  measurement.  It  recommends  that 
agencies  seek  opportunities  to  improve  the  effectiveness  and  efficiency 
of  government  programs  through  work  process  redesign  and  the 
judicious  application  of  IT.  Agencies  must  perform  various  benefit-cost 
analyses  to  support  ongoing  management  oversight  processes  and 
conduct  post-implementation  reviews  of  information  systems.  These 
analytical  reviews  are  used  to  assess  and  document  actual  versus 
estimated  costs  and  benefits,  identify  causes  for  discrepancies,  and  learn 
how  to  manage  IT  more  effectively  in  the  future. 


sAgencies  Annual  Performance  Plans  Under  the  Results  Act:  Art  Assessment  Guide  to 
Facilitate  Congressional  Decisionmaking  (GAO/GGD/AIMD-IO.1.8,  Feb.  1998). 

6See  Executive  Guide:  Effectively  Implementing  the  Government  Performance  and 
Results  Act  (GAO/GGD-96-1 18,  June  1996). 
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The  Clinger-Cohen  Act  requires  agencies  to  establish  goals  for  improving 
efficiency  through  the  effective  use  of  IT.  Performance  measurements 
must  be  established  that  assess  how  well  IT  supports  agency  programs. 
Agency  heads  must  benchmark  agency  process  performance  against 
comparable  processes  in  terms  of  cost,  speed,  productivity,  and  quality 
of  outputs  and  outcomes.  Agency  heads  must  clearly  define  agency 
missions  and  consider  appropriate  process  changes  before  making 
significant  investments  in  IT.  Agencies  must  also  report  annually  on 
operational  improvements  achieved  through  the  effective  use  of  IT.  OMB 
has  issued  specific  guidance  to  assist  agencies  in  the  implementation  of 
the  Clinger-Cohen  Act  that  requests  specific  information  on  IT  cost, 
benefit,  and  risk.  Most  notable  is  OMB  Management  Circular  A-ll,  Part 
3,  that  provides  instructions  on  agency  budget  submissions  and  OMB 
Director's  policy  memorandum  M-97-02  (also  known  as  "Raines'  Rules") 
that  specifies  investment  criteria  that  IT  projects  must  meet  to  be 
included  in  the  President's  budget  submission  to  the  Congress. 

These  various  legislative  and  executive  branch  requirements  create 
pressure  for  top  management's  increased  attention  on  IT  performance. 
However,  for  federal  agency  managers,  the  challenge  is  not  complying 
with  these  legislative  and  regulatory  requirements,  but  "managing  and 
measuring  to  results"  using  a  well-crafted  IT  performance  management 
system.  To  begin,  federal  managers  should  ask: 

•  What  are  enterprise  (agencywide)  and  key  operational  customer  IT 
performance  expectations? 

•  What  are  the  critical  IT  objectives  given  these  expectations? 

•  What  measures  are  appropriate  for  these  IT  objectives? 

•  What  is  IT's  current  baseline  performance  and  what  should  be  the 
target  performance? 

•  How  will  IT  management  and  customers  work  together  to  use  these 
measures  to  leverage  and  improve  IT  performance  in  ways  that  will 
improve  mission  delivery? 

The  thrust  of  these  questions  is  to  determine  how  IT  contributes  to 
achieving  improved  program  outcomes.  Traditional  measures  such  as 
response  time  and  systems  availability  are  by  themselves  insufficient  to 
answer  these  performance  questions.  "Measures  such  as  machine  hours 
are  easy,"  said  one  manager  we  interviewed,  "what  is  difficult  is  how  to 
measure  the  business  value  of  the  applications." 


GAO/AIMD-98-89  IT  Performance  Measurement  Guide 


Page  7 


Identifying  Best  Practices: 

Learning  from  Leading  Organizations 

Federal  managers  are  seeking  guidance  on  developing  and  implementing 
agency  IT  performance  management  systems.7  To  assist  agencies,  we 
examined  how  certain  leading  organizations  approach  IT  performance 
management,  studying  practices  of  both  public  and  private  sector 
organizations  recognized  by  peers  and  independent  researchers  for  their 
IT  performance  efforts.  (A  more  detailed  description  of  our  case  study 
selection  methodology  is  found  in  appendix  2.) 

The  private  sector  companies  used  as  case  studies  were 

•  Xerox  Corporation, 

•  Eastman  Kodak  Company, 

•  Texas  Instruments, 

•  Motorola  Semiconductor  Products  Sector,  and 

•  American  Express  Travel  Related  Services  Company. 

In  the  public  sector,  we  studied 

•  Oregon  Department  of  Transportation, 

•  Sunnyvale,  California,  and 

•  Phoenix,  Arizona. 

We  also  selectively  included  the  U.S.  Immigration  and  Naturalization 
Service,  the  U.S.  General  Services  Administration  Information 
Technology  Service,  and  the  U.S.  Department  of  Agriculture  to  assess 


7  Re  cent  efforts  include  The  Connection:  Linking  IRM  and  Mission  Performance,  a 
resource  paper  sponsored  by  the  Association  for  Federal  Information  Resources 
Management,  September  1995;  Practitioner's  Guide  to  IS.  Performance  Measurement,  a 
guide  issued  by  the  Society  for  Information  Management's  Advanced  Practices  Council, 
March  1995;  Information  Management  Performance  Measures,  a  report  for  the  U.S. 
Department  of  Defense  issued  by  the  National  Academy  of  Public  Administration, 
January  1996,  Performance-Based  Management:  Eight  Steps  to  Develop  and  Use 
Information  Technology  Performance  Measures  Effectively,  a  guide  prepared  by  the 
General  Services  Administration,  December  1996;  and  Guide  for  Managing  Information. 
Technology  fm  as  an  Investment  and  Measuring  Performance.  Department  of  Defense 
guidance  issued  by  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence,  March  3,  1997. 
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federal  practices.8  We  gathered  additional  information  from  general  and 
IT  performance  management  literature  and  reports. 

We  preceded  our  case  study  research  with  an  extensive  review  of 
performance  management  and  measurement  literature,  guides,  and 
reports  published  by  research  groups,  professional  organizations,  and 
academics.  We  also  consulted  with  subject  experts  in  industry  and 
academia  involved  in  IT  performance  management  and  measurement 
research  and  implementation  efforts.  We  collected  case  study  data 
through  interviews  and  documentary  analysis,  not  direct  observation,  and 
we  did  not  validate  the  accuracy  of  this  data.  Case  study  organizations 
reviewed  our  results  for  factual  accuracy  and  completeness.  We  briefed 
numerous  federal  agency  officials  on  our  preliminary  results  to  obtain 
their  views  on  the  applicability  of  our  framework  to  the  federal 
government.  Appendix  II  provides  a  more  detailed  description  of  our 
scope  and  methodology. 

Our  guide  includes  key  practices  identified  in  the  case  studies  and  key 
concepts  and  practices  extracted  from  available  literature,  guides,  and 
reports,  and  supplements  recent  GAO  reports  and  testimonies.9  Without 
exception,  the  organizations  we  studied  noted  that  IT  performance 
management  and  measurement  is  an  evolving  discipline.  As  such,  our 
report  is  neither  complete  or  definitive  and  will  be  refined  and  expanded 
in  the  future. 


8Other  organizations  we  consulted  were  the  Florida  Legislature  Joint  Committee  on 
Information  Technology  Resources,  the  Oregon  Department  of  Administrative  Services, 
the  Oregon  Secretary  of  State  Audits  Division,  the  Office  of  the  Texas  State  Auditor,  the 
Minnesota  Office  of  the  Legislative  Auditor,  the  Minnesota  Department  of 
Transportation,  the  Federal  Emergency  Management  Agency,  the  city  of  Portland,  the 
Treasury  Board  of  Canada,  the  United  Kingdom's  Government  Centre  for  Information 
Systems  and  Royal  Mail,  the  Society  for  Information  Management,  and  the  MITRE 
Corporation. 

9These  include  Government  Reform:  Goal-Setting  and  Performance  (GAO/AIMD/GGD-95- 
130R,  March  27,  1995);  Managing  for  Results:  Experiences  Abroad  Suggest  Insights  for 
Federal  Management.  Reform  (GAO/GGD-95-120,  May  2,  1995);  Managing  for  Results: 
Critical  Actions  for  Measuring  Performance  (GAO/T-GGD/AIMD-95-187,  June  20,  1995); 
Managing  for  Results:  Achieving  GPRA's  Objectives  Requires  Strong  Congressional  Role 
(GAO/T-GGD-96-79,  March  6,  1996);  The  Government  Performance  and  Results  Act: 

1997  Govemmentwide  Implementation  Will  Be  Uneven  (GAO/GGD-97-106,  June  2,  1997); 
Managing  For  Results:  The  Statutory  Framework  for  Improving  Federal  Management 
and  Effectiveness  (GAO/T-GGD/AIMD-97-144,  June  24,  1997). 
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Understanding  the  Context  of  IT 
Performance  Management 

Our  research  shows  that  there  is  no  one  "best"  approach  to  IT 
performance  management.  How  IT  performance  management  is 
designed,  implemented,  and  sustained  in  each  organization  depends  on 
many  factors,  including 

•  the  organization's  culture-whether  the  leadership  supports  IT 
performance  management  and  reflects  this  support  in  its  program  and 
personnel  appraisal  and  reward  systems; 

•  the  importance  of  IT  to  program  (mission)  delivery; 

•  the  accepted  utility  of  IT  in  the  organization; 

•  the  allocation  of  IT  responsibilities  in  the  organization,  including 
whether  IT  is  centrally  managed  or  responsibility  is  dispersed;  and 

•  the  availability  of  resources  (e.g.,  skills,  people,  tools,  money)  to 
support  performance  management. 

Factors  such  as  these,  taken  together,  provide  a  complex  environment 
affecting  IT  performance  management.  The  key  contextual  factor  is  top 
management  ownership,  involvement  in,  and  use  of  IT  performance 
information.  Their  attention,  reinforcement,  and  actions  are  critical 
factors  to  any  organization's  use  of  IT  performance  management. 

In  the  organizations  we  studied,  there  is  commitment  to  IT  measures  and 
their  rigorous  use  in  decision-making  at  all  management  levels  to 
improve  IT  performance.  Moreover,  an  effective  partnership  is  forged 
between  IT  and  the  enterprise  and  operational  customers  so  that  IT 
objectives  and  measures  are  jointly  produced.  Organizational  goals  and 
objectives  drive  IT  goals,  functions,  and  measures.  Organizational 
customers  define  the  mission  objectives  while  IT  management  and  staff 
determine  how  IT  can  best  support  them. 
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Fundamental  Practices:  The  Foundation  of 
IT  Performance  Management 


IT  performance  management  and  measures  are  considered  subsets  of 
overall  performance  management  systems.  In  structuring  an  effective 
approach  to  performance  management,  it  is  also  important  to 

•  differentiate  between  IT's  impact  on  intermediate  versus  final 
program  outcomes, 

•  use  a  good  balance  of  different  kinds  of  IT  measures, 

•  understand  that  measures  may  differ  by  management  tier  within  an 
organization,  and 

•  evaluate  both  the  overall  performance  of  the  IT  function  within  an 
organization  and  the  outcomes  for  individual  IT  investments. 

These  concepts  will  be  discussed  in  greater  detail  in  subsequent  sections 
of  the  guide. 

Our  approach  suggests  three  primary  practice  areas  that  characterize  IT 
performance  management:  aligning  FT  systems  with  agency  missions, 
goals,  and  programs;  constructing  measures  that  determine  how  well  IT 
is  supporting  strategic,  customer,  and  internal  business  needs;  and 
implementing  performance  measurement  mechanisms  at  various 
decision-making  levels  within  an  organization. 

Two  supporting  practice  areas  are  important  to  keep  the  overall  IT 
measurement  process  working.  Data  collection  and  analysis 
capabilities  which  provide  performance  data  that  is  accessible,  reliable, 
and  collected  in  the  least  burdensome  manner.  The  benefit  of  effective 
automated  data  and  management  information  systems  is  that 
performance  information  can  be  effectively  and  efficiently  used  to  make 
strategic,  managerial,  and  day-to-day  operational  decisions.  In  addition, 
a  commitment  to  the  concept  of  continuous  improvement  used  for 
strengthening  the  processes  and  practices  being  used  to  deliver  IT 
products  and  services  is  essential  for  maintaining  effective  IT 
organizations. 

Figure  1  shows  the  generic  model  produced  by  our  case  study  research 
on  IT  performance  measurement. 
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Figure  1:  IT  Performance  Management  Approach 


ALIGN 


REINFORCE 


Practice  Area  1:  Follow  an  IT  "results  chain" 

Leading  organizations  build  and  enforce  a  disciplined  flow  from  goals  to 
objectives  to  measures  and  individual  accountability.  They  define 
specific  goals,  objectives,  and  measures,  use  a  diversity  of  measure  types, 
and  describe  how  IT  outputs  and  outcomes  impact  operational  customer 
and  enterprise  (agency)  program  delivery  requirements.  The  IT 
performance  management  system  does  not  optimize  individual  customer 
results  at  the  expense  of  an  enterprise  (agency)  perspective. 


Practice  Area  2:  Follow  a  balanced  scorecard  approach 

Leading  organizations  translate  organizational  strategy  and  IT 
performance  expectations  into  a  comprehensive  view  of  both  operational 
and  strategic  measures.  Four  generic  goal  areas  include  meeting  the 
strategic  needs  of  the  enterprise,  meeting  the  needs  of  individual 
operational  customers,  addressing  internal  IT  business  performance,  and 
addressing  ongoing  IT  innovation  and  learning. 
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Practice  Area  3:  Target  measures,  results,  and  accountability 
at  different  decision-making  tiers 

For  the  balanced  scorecard  areas,  leading  organizations  match  measures 
and  performance  results  to  various  decision-making  tiers  or  levels. 

These  tiers  include  enterprise  executives,  senior  to  mid-level  managers 
responsible  for  program  or  support  units,  and  lower-level  management 
running  specific  operations  or  projects.  The  organizations  we  studied 
document  goals  and  measures  in  widely  distributed  IT  performance 
improvement  plans.  Individual  appraisals  tie  IT  performance  to 
incentives. 

Practice  Area  4:  Build  a  comprehensive  measure,  data 
collection,  and  analysis  capability 

Leading  organizations  give  considerable  attention  to  baselining, 
benchmarking,  and  the  collection  and  analysis  of  IT  performance 
information.  They  use  a  variety  of  data  collection  and  analysis  tools  and 
methods  that  serve  to  keep  them  informed  but  without  imposing 
unnecessary  reporting  burdens.  They  also  periodically  review  the 
appropriateness  of  their  current  measures. 

Practice  Area  5:  Improve  performance  of  IT  business 

processes  to  better  support  mission  goals 

In  the  leading  organizations,  IT  performance  improvement  begins  and 
ends  with  IT  business  processes.  The  organizations  map  their  IT 
business  processes  and  priortize  among  those  processes  which  must  be 
improved  to  support  an  enterprise  and  operational  customers'  business 
processes. 
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Measurement  Maturity:  Start  With  the  Basics 
and  Increase  Sophistication  Over  Time 

Developing  performance  measures  that  demonstrate  the  impact  of 
information  technology  on  mission  performance  requires  management 
commitment,  experience  in  constructing  and  evaluating  measures,  and  a 
constant  learning  environment.  Many  of  the  organizations  we  talked  to 
indicated  that  had  attempted  to  develop  strategic  or  mission  impact 
measures  without  first  realizing  that  they  had  to  demonstrate  strong 
capability  and  sound  performance  in  the  basics  of  IT  management.  In 
short,  if  an  IT  unit  was  not  being  very  successful  in  providing  quality 
products  and  services  to  the  rest  of  the  organization,  it  had  little 
credibility  in  measuring  strategic  contributions  to  mission  or  business 
results. 

As  such,  several  IT  managers  emphasized  the  need  to  start  with  the 
basics  by  assessing  the  quality  and  effectiveness  of  existing  internal  IT 
operations.  This  evaluation  and  initial  measurement  construction 
exercise  can  focus  on  such  things  as  (1)  delivery  of  reliable,  cost- 
effective,  high  quality  IT  products  and  services,  (2)  adherence  to 
industry  standards  for  systems  design,  cost  estimation,  development,  and 
implementation,  (3)  internal  customer  satisfaction,  (4)  staff  productivity, 
and  (5)  technical  skills  and  capability.  These  factors  are  important  to 
providing  effective  IT  support  to  business  operations  and  improving 
overall  organizational  performance. 

Starting  with  measures  of  internal  IT  operations  offers  some  advantages, 
even  though  they  should  not  be  viewed  as  substitutes  for  other 
measurements  of  IT  contribution  to  specific  program  and  mission  area 
results.  First,  it  preempts  the  problem  of  IT  organizations  waiting  for 
development  and  consensus  of  mission  or  business  specific  performance 
measures.  Second,  it  provides  the  IT  organization  with  valuable 
experience  in  performance  measurement  construction  and  evaluation 
which  is  easily  transferable  to  mission-related  measures.  Third, 
constructing  performance  measures  for  IT  operations  conforms  with  a 
balanced  scorecard  approach  that  emphasizes  the  need  for  a  diversity  of 
measures  in  examining  IT  performance.  And  fourth,  by  introducing 
performance  measurement  maturity  over  time  the  overall  success  of 
implementing  performance  management  in  an  organization  improves 
significantly-an  issue  we  discuss  in  greater  detail  in  the  final  section  of 
this  report. 


Page  14 


GAO/AIMD-98-89  IT  Performance  Measurement  Guide 


IT  Performance  Is  Essential  for 
Strategic  Information  Management 


Performance  measurement  is  not  an  end,  but  rather  the  means  to 
achieving  better  management  results.  In  our  May  1994  Executive  Guide 
on  strategic  information  management,10  we  noted  that  leading 
organizations  use  performance  measures  to  objectively  evaluate  mission, 
business,  and  project  outcomes.  These  organizations  (1)  focused 
performance  measures  on  gauging  service  to  key  management  processes, 
(2)  embedded  performance  measures  in  key  management  processes,  (3) 
used  internal  and  external  benchmarks  to  assess  relative  performance, 
and  (4)  tailored  performance  measures  to  gauge  whether  information 
technology  made  a  difference  in  improving  performance. 

As  shown  in  figure  2  performance  measurement  is  a  cornerstone  practice 
that  GAO  advocates  as  part  of  an  integrated  strategic  information 
management  approach. 

Figure  2:  Performance  Measurement-A  Strategic  Information 
Management  Best  Practice 


Communicate  the  urgency  to  change  IT  practices 
Get  line  management  involved  and  create  ownership 
Take  action  and  maintain  momentum 

Define  customer  needs  and  mission  goals 
Measure  the  performance  of  mission  processes 
Focus  on  process  improvement 
Manage  IT  projects  as  investments 
Integrate  planning,  budgeting,  and  evaluation 

Establish  customer/supplier  relationships  between 
managers  and  IT  professionals 
Create  a  Chief  Information  Officer  position 
Upgrade  IT  skills  and  knowledge 


10GAO/AIMD-94-l  15. 
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In  June  1996,  GAO  issued  a  companion  Executive  Guide  on  a  suggested 
performance  measurement  implementation  approach  for  the  Results 
Act.11  The  approach,  depicted  in  figure  3  identifies  certain  key  steps  and 
associated  practices  that  agencies  may  find  useful  for  implementation  of 
the  Results  Act.  The  approach  is  based  on  the  actions  taken  by  certain 
organizations  that  have  successfully  undertaken  performance 
improvement  initiatives  similar  to  that  required  by  the  act. 

Figure  3:  Implementing  the  Results  Act— Key  Steps  and 
Critical  Practices 


STEP  3 


Use  Performance 
Information 

Practices: 

6.  Identify  performance 
gaps 

7.  Report  information 

8.  Use  information 


STEP  1 

Define  Mission  and 
Desired  Outcomes 
Practices: 

1.  Involve  stakeholders 

2.  Assess  environment 

3.  Align  activities, 
core  processes, 
and  resources 


Reinforce  GPRA 
Implementation 

Practices: 

9.  Devolve  decision-making 
with  accountability 

10.  Create  incentives 

11.  Build  expertise 

12.  Integrate  management 
reforms 


STEP  2 


Measure  Performance 

Practices: 

4.  Produce  measures  at 
each  organizational 
level  that 

•  demonstrate  results, 

•  are  limited  to  a  vital 
few, 

•  respond  to  multiple 
priorities,  and 

•  link  to  responsible 
programs 

5.  Collect  data 


Each  organization  GAO  studied  set  its  agenda  for  management  reform 
according  to  its  own  environment,  needs,  and  capabilities.  In  striving  to 
become  more  results  oriented,  they  commonly  took  three  steps  in 
implementing  a  performance  based  management  approach.  First,  they 
defined  clear  missions  and  desired  outcomes.  Second,  they  measured 


nGAO/GGD-96-l  18. 
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performance  to  gauge  progress.  Third,  they  used  performance 
information  as  a  basis  for  decision-making. 

Along  with  these  steps,  certain  practices  proved  especially  important  to 
the  success  of  their  efforts.  Taken  together,  these  steps  and  practices 
were  useful  in  making  changes  necessary  for  these  organizations  to 
become  results  oriented.  These  fundamental  steps  and  practices  are 
consistent  with  the  Results  Act  requirements  and  provide  a  useful 
framework  for  federal  agencies  to  adopt  in  implementing  key  provisions 
of  the  law. 

The  IT  performance  management  approach  outlined  is  this  guide  works 
in  tandem  with  GAO's  Results  Act  implementation  model  and 
demonstrates  how  IT  performance  measurement  can  be  implemented 
within  an  overall  performance  management  framework.  Most 
importantly,  mission  goals,  objectives,  and  strategies  must  be  understood 
in  order  to  evaluate  how  IT  contributes  to  performance  improvements. 

In  the  sections  that  follow,  each  practice  area  in  our  approach  is 
explained  in  detail,  listing  specific  characteristics  and  providing  case 
study  examples  to  illustrate  how  they  are  implemented.  A  final  section 
discusses  key  steps  involved  in  implementing  an  IT  performance 
management  system. 
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Practice  Area  1: 
Follow  an 
Information 
Technology 
"Results  Chain" 


ALIGN 


IMPLEMENT  . 


CONSTRUCT 


Practice  Area  4 


practice  Area  3\ 

Build  data 

/Practice  Area  2\ 

/  Target  measures  ’ 

i  collection/analysis 

■  Follow  a  \ 

1  at 

capability  j 

1  balanced  1 

1  decision-making  j 
V  levels  J 

Practice  Area  5 

1  scorecard  J 

\  approach  J 

Improve  IT 
processes 

REINFORCE 

Practice  Area  Characteristics 

1.  Directly  map  IT  goals  and  measures  to  organizational  mission 
goals,  objectives,  and  measures 

2.  Prepare  a  chain  of  events  and  evidence  to  understand  IT's 
contribution  to  enterprisewide  and  operational  customer 
objectives 

3.  Use  a  diversity  of  measures  to  evaluate  IT  performance 


Practice  Area  Overview 

To  maximize  the  results  of  IT  investments,  leading  organizations  ensure 
that  IT  programs  align  with  and  directly  support  high-level  organizational 
missions,  goals,  and  objectives.  This  practice  provides  an  approach  for 
linking  organizational  goals  and  objectives  to  the  "vital  few"  IT 
performance  measures  needed  to  manage  for  effective  results.  This 
framework,  formal  or  informal,  follows  a  systematic  movement  through 
what  can  be  called  an  "IT  results  chain."  The  results  chain  approach 
provides  discipline  for  aligning  performance  expectations  and  measures 
at  all  levels. 

Any  effort  to  measure  IT  performance  must  begin  with  clearly  defined 
organizational  and  programmatic  goals  and  objectives.  In  other  words, 
an  organization  cannot  properly  define  its  IT  goals  and  objectives  (much 
less  measure  to  determine  the  degree  of  success  in  meeting  them)  unless 
it  has  clearly  defined  goals  and  objectives  for  the  programs  that  IT 
supports.  The  resulting  IT  goals  and  measures  must,  in  all  cases,  map 
back  to  program  or  strategic  (enterprise-level)  goals.  To  help  understand 
the  relationships  between  operational  programs  and  the  IT  contribution 


Page  18 


GAO/AIMD-98-89  IT  Performance  Measurement  Guide 


to  their  success,  many  organizations  prepare  a  chain  of  events  and 
evidence  to  show  how  programs  work  and  how  success  might  be 
measured.  Finally,  a  diversity  of  qualitative  and  quantitative  measures 
are  used  for  the  inputs,  outputs,  and  outcomes  of  IT  programs. 

In  short,  a  results  chain  approach 

•  defines  what  the  organization  is  attempting  to  accomplish, 

•  allows  an  organization  to  identify  success,  and 

•  links  IT  projects  directly  to  business  goals  and  objectives. 

As  shown  in  figure  4,  the  chain  shows  the  finks  from  organizational  goals 
and  objectives  to  IT  performance  measures.  In  the  organizations  we 
studied,  measuring  IT's  contribution  begins  by  defining  organizational 
goals  and  objectives--for  the  enterprise  and  for  internal  and  external 
customers.  These  goals  and  objectives  should  be  based  on  defined 
organizational  mission  statements. 

Figure  4:  An  IT  Results  Chain 
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As  for  other  programs,  IT  management  and  staff  can  then  develop  a 
purpose  statement  that  specifically  defines  how  IT  products  and  services 
will  be  used  to  support  the  achievement  of  organizational  and  customer 
goals.  The  purpose  statement  is  then  translated  into  IT  goals,  objectives, 
and  measures. 

IT  goals  and  objectives  should  be  consistent  with  the  IT  purpose 
statement  and  clearly  linked  to  organizational  goals  and  objectives. 
Leading  organizations  focus  on  a  "vital  few  IT  objectives"  to  demonstrate 
results  in  selected  key  areas.  Similarly,  the  number  of  measures  for  each 
IT  goal  should  be  limited  to  the  "vital  few."  These  should  be  limited  to 
the  key  IT  performance  dimensions  that  will  enable  the  IT  organization 
to  assess  accomplishments,  make  decisions,  realign  processes,  and  assign 
accountability.  Attempts  to  manage  an  excess  number  of  IT  measures 
increase  risks  of  confusing  excess  data  with  key  IT  performance  issues. 
Lastly,  management  and  staff  performance  evaluations  are  linked  to  the 
performance  measures  as  a  way  of  comparing  achievements  with 
planned  results. 

As  part  of  a  results-oriented  management  approach,  IT  performance 
measurement  must  be  used  in  the  decisionmaking  process.  Measurement 
development  and  alignment  involves  consideration  of  all  organizational 
goals  and  objectives  and  converging  on  the  vital  few  IT  goals,  objectives, 
and  measures.  Effective  measurement  must  be  supported  by  sound  data 
collection  and  analysis  methods  and  communication  of  results  to 
management  and  staff. 

Practice  Area  Characteristics 

1.  Directly  Map  Information  Technology  and  Management  Goals 
and  Measures  to  Strategic  Goals 

Use  of  an  IT  results  chain  is  only  as  good  as  the  clarity  and  specificity  of 
the  overall  organizational  goals  and  objectives.  Leading  organizations 
build  consensus  among  program  managers,  IT  managers,  customers, 
stakeholders,  and  staff  to  establish  joint  ownership  for  performance 
management.  They  work  together  to  achieve  a  common  understanding 
of  goals,  objectives,  measures,  and  anticipated  outcomes.  As  a  practical 
matter,  those  who  will  judge  the  success  of  programs  and  the  supporting 
functions  should  agree  on  the  links  in  the  results  chain  from  IT's  purpose 
to  the  vital  few  measures.  11 


aFor  related  guidance,  see  Agencies'  Strategic  Plans  Under  GPRA  Key  Questions  to 
Facilitate  Congressional  Review  (GAO/GGD-IO.1.16,  May  1997). 


Page  20 


GAO/AIMD-98-89  IT  Performance  Measurement  Guide 


In  the  organizations  we  examined,  IT  goals  and  measures  flow  directly 
from  strategic  goals.  IT  managers  and  staff  do  not  develop  performance 
management  systems  that  optimize  operational  customer  results  without 
considering  an  enterprisewide  perspective.  IT  goals  and  measures  in 
support  of  individual  operational  customers  must  meet  IT  department  or 
unit  objectives.  In  turn,  IT  department  or  unit  objectives  must  map 
directly  to  both  programmatic  and  enterprisewide  strategic  directions  or 
goals.  The  result  is  that  IT  goals  and  measures  track  in  a  seamless 
fashion  back  to  enterprise  strategic  directions  or  goals.  If  such  mapping 
is  not  obvious  when  comparing  measures  and  high-level  goals,  the  IT 
function  is  probably  not  measuring  the  right  things. 


Case  Study  Is 

Linking  Measures  to  Specific  IT  Goals  and  Objectives 

The  city  of  Phoenix's  Information  Technology  Department  (ITD)  wanted  a  set  of  business-oriented 
customer-based  measurements  to  measure  the  effectiveness  of  its  ITD  operations .  One  manager  said,  "We 
set  forth  to  come  up  with  a  ' net  result  measuring  scheme . '  It  would  (1)  focus  on  customer  satisfaction,  (2) 
be  employee  friendly,  (3)  not  affect  program  work,  (4)  improve  customer  satisfaction,  and  (5)  give  the 
customer  the  ability  to  understand  the  value  our  organization  gives  to  them,  and  let  them  suggest  how  to 
help  us  get  better . " 

The  department  wanted  to  ( I)  lay  the  groundwork  for  continuous  monitoring  and  improving  customer 
satisfaction  with  ITD  service,  (2)  better  define  the  value-added  of  ITD  services  from  a  business 
perspective,  (3)  identify  continuous  improvement  needs  in  internal  ITD  operations,  and  (4)  strengthen  ITD 
relationships  with  other  city  departments  and  manage  customer  expectations  using  hard  facts  and 
measures. 

The  department  identified  measures  based  on  input  from  key  selected  customers  and  staff  and  the  ITD 
objectives.  Objectives  were  developed  using  the  City  of  Phoenix  Strategic  Directions  Report  and  the  IT 
architecture  vision.  Generally,  the  strategic  directions  were  to  (I)  create  and  maintain  a  well-informed 
and  involved  community  with  a  clean  and  safe  environment,  efficient  transportation  systems,  a  quality 
education  system,  and  economic  opportunity  for  all  citizens,  (2)  provide  cost-effective,  high-quality 
community  services  through  a  productive,  efficient  organization  and  an  empowered  workforce,  and  (3) 
generate  and  maintain  desirable  revenue  flows  and  sound  financing  within  a  system  of  rational  resource 
allocation.  Goals  for  the  architecture  vision  were  to  make  all  applications  integrated  and  fully 
compatible,  using  a  city-wide  deployed  IT  architecture  supported  by  commonly  shared  data  repositories. 
The  department  then  revised  the  measures  to  meet  all  of  the  ITD  objectives  and  correlated  the  measures 
to  customer-defined  goals  of  time,  cost,  and  customer  satisfaction. 

The  table  on  the  following  page  shows  how  Phoenix  measures  progress  towards  achieving  department 
objectives.  These  measures  focus  on  timeliness  (service  delivery),  cost  (producing  city  services  at 
acceptable  or  reduced  cost),  or  customer  satisfaction  (evidence  that  internal  and  external  customers  are 
satisfied  and  that  ITD's  contribution  can  be  quantified). 
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Case  Study  1: 

The  City  of  Phoenix:  Relating  Performance  Metrics  to  IT  Objectives 


IT  Department 

Objectives  for  Meeting 

City  Strategies 

Net  Results  Metrics 

Used  for  IT  Objectives 
( Examples ) 

IT  Objectives 
Related  to  the 
Metrics  (see 
column  one) 

Type  of 

Measure 

i 

Build  partnerships  with 
city  departments. 

Successful  delivery  of  ITD  products 
or  services  on  time  v.  goal. 

2,  3,  4,  5,  6,  11 

Time 

2 

Provide  enabling 
technology  to  city 

Reduced  delivery  time  v.  goal. 

3,  4,  5,  6,  11 

Time 

departments. 

Problem  responsiveness  v.  goal. 

1,  3,  4,  5,  6,  8,12 

Time 

3 

Increase  customer  \ 

satisfaction. 

Reduction  attributed  to  ITD  v.  goal. 

3,  9,  10,  11,  12 

Cost 

4 

Reduce  cycle  times  in 
delivery  of  ITD 
products  and  services. 

Customer  satisfaction  and 
relationships  v.  goal. 

all 

Customer 

satisfaction 

5 

Reduce  service 
delivery  costs. 

Reliability  of  products  and  services 
v.  goal 

3,  6,  8,  12 

Customer 

Satisfaction 

6 

Improve  service 
delivery  processes. 

Systems  using  citywide 
architecture/new  emerging 
technologies  v.  goal. 

1.3,  4,  7,  8,  9,  10 

Customer 

Satisfaction 

7 

Implement  citywide 

technology 

architecture. 

ITD  staff  trained  in  customer 
service  skills  and  new  technology  v. 
goal. 

1,  2,  3,  9,  10,  11 

Customer 

Satisfaction 

8 

Improve  citizen  access 
to  information  and 
services  through 

New  ideas  received  and  adopted 
from  ITS  staff  v.  goal. 

1,  3,  6,  9,  10,  11 

Customer 

Satisfaction 

technology. 

Employee  satisfaction  v.  goal. 

3,  9,  10,  11 

Customer 

Satisfaction 

9 

Increase  resource 
versatility. 

Effective  communications  v.  goal. 

1,  2,  3,  4,  11,  12 

Customer 

Satisfaction 

10  Improve  leadership 
skills. 

ITD  involvement  in  departmental 
technology  planning  v.  goal. 

1,  2,  3,  4,  7,  11,  12 

Customer 

Satisfaction 

11  Increase  employee 
confidence. 


12  Achieve  world  class 
results 
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Case  Study  2: 

Connecting  Goals,  Strategies,  Objectives  and  Measures 

To  better  meet  customer  needs  and  respond  to  a  changing  market  environment ,  in  the 
mid  1990' s  Xerox  developed  a  Xerox  2000  business  strategy .  In  response  to  the  new 
strategy },  the  information  management  organization-Global  Process  and  Information 
Management  (GP&IM)- -developed  four  strategies  and  three  breakthrough  goals  to  align 
IM  programs  to  support  achievement  of  the  outcomes  anticipated  by  the  corporate  level 
business  drivers.  Senior  executives  ultimately  decided  on  eight  measures  to  determine 
whether  IM  programs  were  supporting  corporate-level  strategies. 

GP&IM  concluded  that  five  factors  were  significantly  influencing  how  it  developed  a  new 
IT  strategy  for  Xerox.  These  were  (1)  the  push  to  reengineer  Xerox  business  processes,  (2) 
knowledge  that  the  current  information  management  environment  was  not  the 
foundation  for  the  future  of  Xerox,  (3)  information  systems  investment  decisions  were 
currently  driven  by  " entitlements , "  not  a  strong  selection  strategy,  (4)  renewal  of  the 
existing  infrastructure  was  too  costly,  and  (5)  the  cycle  time  for  information  management 
was  not  meeting  business  requirements. 

GP&IM  then  developed  four  strategies  to  respond  to  these  five  IT  strategy  "drivers":  (1) 
reduce  and  redirect  information  spending  through  global  outsourcing,  reducing  legacy 
system  spending,  and  consolidating  and  sharing  resources,  (2)  infrastructure 
management  including  asset  management,  (3)  leverage  worldwide  information  resources 
of  people,  hardware,  and  software,  and  (4)  develop  business  process-driven  solutions. 

As  shown  in  the  figure  on  the  following  page,  GP&IM  has  several  outcomes  identified  for 
each  of  the  four  strategies.  GP&IM  also  developed  three  proposed  high-level 
"breakthrough"  goals  covering  the  four  strategies,  shown  on  page  24.  The  goals  are 
defined  as  follows: 

Manage  IM  spending  suggests  that  IM2000  must  be  to  managed  to  a  flat  ceiling  for 
base  spending  while  doubling  the  amount  of  investment  in  new  development. 

Deliver  infrastructure  renewal  determines  the  penetration  of  the  new  infrastructure  in 
conjunction  with  some  renewal  of  the  old  infrastructure  (known  as  Globalview). 

Deliver  business  process  solutions  covers  reengineering  process  deployment  and  drives 
operational  optimization  and  the  development  of  a  baseline  for  day-to-day  work.  The 
baseline  is  for  comparison  with  Electronic  Data  Systems  (EDS)  outsourcing 
arrangements.  Each  breakthrough  goal  area  has  measure  examples. 

Xerox  top  management  decided  that  the  eight  measures  cited  in  figure  8  provided  the  right 
information  on  whether  their  IT  strategies  were  being  achieved.  For  example ,  the  measures  told 
them  how  well  they  were  performing  in  actually  redirecting  IT  spending  into  new  applications  and 
systems  development ,  retiring  the  old  infrastructure  and  replacing  it  with  more  companywide 
solutions ,  reducing  operations  and  maintenance  costs ,  and  whether  new  process-driven  IT  solutions 
were  providing  timely ,  high-quality  products  near  estimated  costs. _ _ _ _ 
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Case  Study  2 

Xerox  IM  2000:  Connecting  Strategies  With  Outcomes 


Strateaies 

Expected  Outcomes 

_ i 

r 

•  Orderly  retirement  of  older  infrastructure  and  applications 

•  Increased  IT  investment  into  new  applications  and  infrastructure 
development  and  deployment 

•  Smaller  Xerox  IM  staff  focused  on  high  value  IT  projects 

Reduce/Redirect  ll\/h\ 
Spending 

- 1 

V _ . 

_ CT\ 

/'  ... 

•  Dollar  savings  from  reduction  in  acquisition  costs 

1 

Infrastructure  '\> 

Management 

•  Better  workstations  and  productivity  tools 

•  Movement  from  central  to  distributed  processing 

•  Increased  interoperability 

•  Increased  use  of  company-wide  software/applications 

V _ _ _ _ _ 

- 

_ y 

_ rT\ 
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•  Better  alignment  of  IT  spending  with  business  priorities/needs 

•  Increased  IM  productivity 

•  Improved  IT  resource  allocation  via  a  new  solutions  delivery 
process 

\ . . 

~~~\ 

i 

Leverage  Worldwide^ 

IM  Resources 

- 

_ rT\ 

•  IT  resources  focused  on  business  process  needs 

•  Increased  speed  in  delivering  new  solutions 

•  Tighter  IT  spending  focus 

•  Old  applications  replaced  and  legacy  spending  eliminated 

L  .  _  _ _ 

1 

Business  Process^\ 
Driven  Solutions 

- 
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Case  Study  2: 

Xerox  IM2000:  Breakthrough  Goals  and  Related  Measures 


IM2000  Strategies 

1 - 

Reduce/Redirect  IM 

Spending  / 


Infrastructure 

Management 


Breakthrough  Goals  Measures  Used 
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>  Deliver  Process 
>Driven  Solutions 


IM  spending  as  %  of 
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%  [investment  fund/total 
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•  %  [new  infrastructure 
installed/total  eligible 
sites] 

•  %  old  infrastructure 
removed 

•  %  [approved  reengineered 
processes  deployed/total 
business  processes] 

•  %  compliance  to 
applications  retirement 
plan 

•  %  reuse  code  for  in-house 
deployed  applications 

•  %  compliance  to  approved 
solution  deployment 
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2.  Prepare  a  Chain  of  Events  and  Evidence 

A  common  measurement  problem  is  determining  what  impact  support 
functions  such  as  IT  have  on  operational  customer  or  enterprise 
outcomes  in  comparison  to  other  factors.  Since  final  enterprise  or 
program  outcome  measures  are  difficult  to  measure  for  a  support 
function  like  IT,  it  is  useful  to  think  in  terms  of  a  "chain  of  events  and 
evidence." 

If  given  careful  consideration,  a  chain  of  events  and  evidence  can  help  IT 
managers  understand  how  IT  resources  are  being  used  to  support 
mission  or  specific  program-related  outcomes.  In  some  cases,  IT  is  not 
directly  related  to  final  program  results.  However,  there  are  intermediate 
outcomes  that  must  occur  before  the  program  or  business  outcome  can 
be  achieved.  In  contemporary  organizations,  IT  can  play  a  pivotal  role  in 
achieving  these  intermediate  outcomes. 

A  chain  of  events  is  a  theory  of  how  inputs  in  an  action  chain  move  to 
outputs  and  finally  to  end  outcomes.  A  corresponding  chain  of  evidence 
shows  the  evidence  and  measures  that  match  to  the  chain  of  events.  The 
chains  of  events  and  evidence  recognize  that  agency  programs  set  in 
motion  a  sequence  of  events  which  are  expected  to  achieve  desired 
goals.  The  sequence  of  events  bridges  inputs  to  outputs  to  outcomes. 

The  hypothetical  example  presented  in  figure  5  illustrates  a  simple  chain 
of  events,  evidence,  and  measures  for  an  environmental  water  quality 
program.  At  the  beginning  point  of  the  program  results  chain,  a  series  of 
simple  events  are  mapped  out  to  indicate  steps  essential  to  achieving  the 
end  outcome  of  a  desired  level  of  environmental  quality.  Certain  types 
of  evidence  would  then  be  used  as  indicators  of  whether  the  steps  are 
occurring.  IT  provides  essential  support  for  several  of  the  steps  in  the 
results  chain  which  can  indirectly  affect  the  achievement  of  the  program 
outcome.  Measuring  how  well  IT  is  supporting  these  intermediate  steps 
is  necessary  to  demonstrate  how  well  it  is  performing  from  a  mission 
perspective. 
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Figure  5:  A  Hypothetical  Chain  of  Events/Evidence  for  Achieving 
Environmental  Quality 
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In  an  iterative  process,  organizations  examine  the  chain  of  events  and 
evidence  for  enterprise  and  operational  customer  goals  and  objectives. 
The  chains  explain  how  an  enterprise  or  operational  customer  produces 
its  results  and  just  what  those  results  are.  The  assumption  in  using  the 
chains  of  events  and  evidence  is  that  there  is  a  clear  understanding  of 
just  how  a  program,  for  example,  is  supposed  to  work  in  producing  the 
expected  results. 

In  turn,  a  support  function  such  as  IT  needs  to  define  its  own  chains  of 
events  and  evidence,  demonstrating  how  its  products  and  services  affect 
enterprise  and  operational  customer  chains.  For  IT,  the  operational 
customer  is  interested  in  having  IT  applications  that  help  achieve 
efficient  and  effective  program  operations  and  service  delivery.  In 
essence,  final  IT  outcomes  are  often  efficient  and  effective  program 
operations-  actually  an  operational  customer's  goal.  The  organization  in 
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large  part  gauges  IT  failure  or  success  by  how  well  IT  supports  the  chain 
of  events  the  operational  customer  has  in  place  to  achieve  its  desired 
effect. 

Building  chains  of  events  and  evidence  in  partnership  with  enterprise  and 
organizational  customers  can  be  a  difficult  process,  but  it  helps  IT 
managers  understand  exactly  how  IT  products  and  services  support 
customers  and  how  IT  performance  should  be  measured.  Using  a  chain 
of  events  approach  also  enhances  customer  understanding  of  just  how  IT 
can  contribute  to  the  eventual  enterprise  or  program  outcome. 

3.  Use  a  Diversity  of  Measures  to  Evaluate  IT  Performance 

An  effective  IT  performance  management  system  should  have  a  diversity 
of  measures,  matched  to  the  right  organizational  need  and  level  of 
decision-making,  and  action  taken  on  the  measure.  These  measures  can 
capture  performance  at  the  individual,  organizational,  program,  and 
process  levels.  Generically,  measures-both  qualitative  and  quantitative- 
are  often  categorized  into  four  main  types: 

■  Input  Measures  are  assessments  of  the  resources  used  to  carry  out 
a  program  or  activity  over  a  given  time  period  with  the  purpose  of 
achieving  an  outcome  or  output.  Input  measures  can  include  number 
of  IT  managers  and  employees,  labor  hours,  IT  funds,  computer  and 
telecommunications  equipment  or  facilities,  or  supplies. 

■  Output  Measures  are  assessments  of  the  actual  level  of  work 
accomplished  or  services  provided  over  a  given  time  period.  They  are 
often  used  to  control  resources.  Number  of  reports  issued,  number  of 
projects  completed,  number  of  answers  on  hot  line  services,  and 
function  points  delivered  are  examples  of  output  measures.  These, 
too,  are  often  process  measures. 

■  Outcome  Measures  assess  the  actual  results,  effects,  or  impacts  of  a 
program  or  support  function  compared  to  its  intended  purpose. 
Outcomes  can  be  difficult  to  measure  because  results  may  not  be 
immediately  evident  or  several  organizational  units  and  external 
suppliers  or  customers  are  involved  and  it  is  difficult  to  assign  relative 
contributions  to  them.  Outcome  measures  may  be  the  level  of 
customer  satisfaction  with  IT  services  or  cycle  time  reduction 
attributable  to  automated  work  processes. 

■  Combinations  of  Single  Measures  combine  single  output,  outcome, 
and/or  input  measures  into  measures  designed  to  demonstrate 
improvements  in  efficiency  or  effectiveness.  An  efficiency  measure  is 
output  over  input,  such  as  number  of  PC  applications  installed  per 
number  of  IT  staff.  An  effectiveness  measure  may  compare  actual 
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results  to  estimated  or  expected  results  or  compare  existing  levels  of 
performance  (output)  to  accepted  industry  standards  or  target 
performance  goals. 

A  wide  range  of  measures  provide  a  balance  for  different  decisionmaker 
needs.  Input  and  output  measures  assess  workload  for  an  enterprise  or 
specific  program  and  how  much  in  demand  are  its  products  and  services. 
Combination  measures  assess  efficiency  and  effectiveness.  Outcome 
measures  assess  results  compared  to  expectations.  The  key  point  is  that 
the  right  measure  is  used  at  the  right  time  and  for  the  right  reason. 

Input  and  output  measures  are  absolutely  vital  for  measuring  how  well  a 
process  to  deliver  IT  products  and  services  is  performing. 

Use  Contextual  Information  to  Augment 
Performance  Measures 

The  organizations  we  studied  track  contextual  or  explanatory 
information  to  use  with  their  IT  performance  measures.  The  contextual 
or  explanatory  information  describes  the  broader  environment  that 
surrounds  IT  activities  which  can  influence  IT  inputs,  outputs,  and 
outcomes.  For  example,  measuring  changes  in  mainframe  to  client 
server  use,  policy  changes  that  impact  performance,  or  changes  in  IT 
organizational  structure  could  be  important  explanatory  information. 

Capturing  contextual  and  explanatory  information  can  help  managers 
understand  the  measures,  assess  performance,  and  evaluate  the 
significance  of  underlying  factors  that  may  affect  reported  performance. 
Managers  can  also  capture  actions  that  have  been  taken  or  are  being 
taken  to  in  response  to  reported  information,  particularly  for 
unexpectedly  high  or  low  performance.  Often,  IT  performance  data 
displays  show  the  contextual  or  explanatory  information  either  as  a 
footnote  to  the  performance  data  or  in  an  explanatory  page  attached  to 
the  performance  information. 
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How  to  Get  Started 


To  proceed  with  the  development  and  use  of  an  IT 
approach,  organizations  should 


results  chain 


clarify-with  top  management  participation  and  stakeholder 
involvement-major  organizational  goals  and  objectives; 


establish  a  simple  chain  of  events  and  evidence  for  key  mission 
areas  of  the  organization; 

create  supporting  IT  outcome  and  process  measures  for  each 


organizatioi 

ial  mission  goal  area; 

•  use  one  of  i 

the  organizational  goals  and  objectives  to  develop  IT 

goals,  spec 
for  that  IT  { 

ific  objectives,  and  related  performance  measures 
*oal;- 

♦  examine  ex 
or  outcome 
measures  tl 

isting  IT  measures,  categorize  them  as  input,  output, 
measures,  and  decide  on  the  combination  of 
tat  best  provides  performance  results;  and 

•  test  the  per 

formance  measurement  system  and  make  revisions 

based  on  initial  lessons  learned. 
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Practice  Area  2: 
Follow  a  "Balanced 
Scorecard"  Approach 


ALIGN 


REINFORCE 


Practice  Area  Characteristics 

1.  Develop  IT  goals,  objectives,  and  measures  in  operational 
and  strategic  areas 

2.  Focus  on  the  most  important  "vital  few"  objectives  and 
measures  in  four  IT  goal  areas: 

•  Achieving  the  strategic  needs  of  the  enterprise 

•  Satisfying  the  needs  of  individual  customers 

•  Fulfilling  IT  internal  business  performance 

•  Accomplishing  IT  innovation  and  learning 


Practice  Area  Overview 

A  second  best  practice  is  to  use  a  balanced  scorecard  approach  to  IT 
performance  measurement.  The  approach  attempts  to  create  a 
measurement  balance  across  the  overall  performance  management 
framework.  A  balanced  approach  to  measuring  the  contribution  of  IT  to 
mission  outcomes  and  performance  improvement  recognizes  the  broad 
impact  of  IT's  supporting  role.  By  measuring  IT  performance  across  four 
goal  areas  that  are  critical  to  overall  IT  success,  the  scorecard  forces 
managers  to  consider  measurement  within  the  context  of  the  whole 
organization.  This  limits  the  possibility  of  overemphasizing  one  area  of 
measurement  at  the  expense  of  others.  In  addition,  measuring  IT 
performance  from  different  perspectives  helps  strengthen  the  analysis  of 
intangible  and  tangible  benefits  attributable  to  technology. 
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In  the  four  IT  goal  areas  discussed  in  this  section,  we  present  three  or  four 
key  objectives  that  were  common  among  the  organizations  we  examined. 
Corresponding  to  each  objective  we  provide  some  sample  measures  that 
come  from  our  case  study  research  and  from  supporting  literature.  Our 
purpose  is  to  illustrate  possible  types  of  measures,  not  to  prescribe  a 
definite  list  of  measures  that  all  organizations  should  be  using.  Some  of  the 
measures  are  very  basic,  but  they  are  clearly  related  to  the  objectives.  Also, 
many  of  the  measures  are  percentages  or  ratios.  This  is  important  because 
successful  organizations  begin  with  good  baseline  data  on  performance  and, 
therefore,  can  accurately  measure  progress  against  the  baseline  as  they 
move  forward. 

In  several  of  the  organizations  we  studied,  management  is  developing 
measures  across  key  areas  covering  both  long-  and  short-term  strategies  and 
activities.  This  approach  is  best  captured  in  Robert  Kaplan  and  David 
Norton's  balanced  scorecard  approach,  which  many  of  the  organizations 
either  used  directly  or  incorporated  into  the  development  their  own 
approaches.11  The  Kaplan  and  Norton  scorecard  evaluates  performance  in 
four  areas:  financial  (how  does  the  organization  look  to  shareholders?), 
customer  (how  do  customers  see  performance?),  internal  business  (at  what 
must  the  organization  excel?),  and  innovative  and  learning  (can  the 
organization  continue  to  improve  and  create  value?). 

In  order  to  summarize  the  IT  performance  methods  being  used  by  the 
organizations  we  studied,  we  have  adopted  a  balanced  scorecard  approach 
similar  to  the  Kaplan  and  Norton  framework.  However,  a  balanced 
scorecard  is  just  one  approach  available  for  agencies  to  adopt  in  conducting 
IT  performance  management  and  measurement.  Other  approaches  such  as 
the  value  management  framework,  critical  success  factor  analysis,  and 
information  economics  also  offer  useful  IT  performance  measurement 
methodologies.12  Like  other  methodologies,  a  balanced  scorecard  approach 
translates  organizational  strategy  into  specific  measurable  objectives, 
operating  from  several  key  concepts: 

•  no  single  measure  provides  clear  performance  targets  or  places  attention 
on  critical  mission  areas, 

•  goal,  objective,  and  measure  areas  should  give  a  comprehensive  view  of 
all  levels  of  activities,  from  the  project  level  to  the  strategic  level, 

•  limiting  the  number  of  measures  used  minimizes  information  overload, 
and 


nKaplan  and  Norton  published  a  series  of  articles  in  the  Harvard  Business  Review  which  explain 
the  concept  and  its  application.  The  articles  are  listed  in  the  bibliography. 

12A  complete  description  of  IT  performance  measurement  approaches  can  be  found  in  the  Society 
for  Information  Management's  Practitioner's  Guide  to  I.S.  Performance  Measurement,  referenced  in 
appendix  I. 
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•  a  scorecard  guards  against  optimizing  one  goal  area  at  the  expense  of 
others. 


Practice  Area  Characteristics 

1.  Develop  IT  Goals,  Objectives,  and  Measures  in 
Operational  and  Strategic  Areas 

For  IT,  measures  cover  a  great  diversity  of  value-added  activities,  including 
those  for  projects,  a  portfolio  of  applications,  and  infrastructure 
development.  Organizations  should  know  about  success  in  all  of  them.  As 
shown  in  figure  6,  an  IT  results  chain  can  be  translated  into  a  scorecard 
framework  that  looks  at  goals,  objectives,  measures  (tiered  for  various 
decision-making  levels),  and  accountability  in  key  goal  areas.  The  key 
starting  point  in  developing  a  balanced  scorecard  is  the  question  of  purpose 
from  the  IT  results  chain-"What  is  the  current  and  future  purpose  of  IT? 
Then  to  meet  that  purpose,  the  IT  organization  must  answer  the  goal 
question--"If  we  succeed,  how  will  we  differ?"  in  terms  of  specific  goals. 

Figure  6:  An  IT  Balanced  Scorecard  Approach 
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The  IT  goals  and  objectives  of  the  organizations  we  studied  most  often 
focused  on  the  following: 

•  customer  commitments  and  satisfaction, 

•  cycle  and  delivery  time, 

•  quality, 

•  cost, 

•  financial  management, 

•  IT  infrastructure  availability, 

•  internal  IT  operations, 

•  IT  skill  availability,  and 

•  customer  business  process  support. 

For  example,  Motorola's  Semiconductor  Products  Sector  (SPS)  focuses  its 
goals  and  objectives  on  four  areas:  (1)  delivering  reliable  products  (quality 
environment,  best-in-class  staff,  worldwide  resource  optimization, 
communication),  (2)  providing  integrated  IT  solutions  (integrated  data  and 
systems  architecture,  technology  roadmap  and  migration  planning, 
distributed  computing  effort),  (3)  building  client  partnerships  (client 
involvement,  lead  the  information  technology  community),  and  (4)  achieving 
competitive  advantage  (prioritize  the  projects  that  benefit  SPS,  deploy 
resources  for  maximum  impact,  speed  of  execution). 

We  developed  four  balanced  scorecard  goal  areas,  objectives,  and  measures 
for  IT  that  were  among  the  most  common  across  the  organizations  we 
studied.  As  such,  the  four  goal  areas  illustrate  an  approximate  consolidation 
of  the  performance  management  efforts  of  the  organizations  involved  in  our 
research.  The  four  balanced  scorecard  goal  areas  are  designed  to  measure 
how  well  IT  is 

•  achieving  the  strategic  needs  of  the  enterprise  as  a  whole,  in  contrast  to 
specific  individual  customers  within  the  enterprise, 

•  satisfying  the  needs  of  individual  customers  with  IT  products  and 
services, 

•  fulfilling  internal  IT  business  performance  that  delivers  IT  products  and 
services  for  individual  customers  and  the  enterprise,  and 

•  accomplishing  ongoing  IT  innovation  and  learning  as  IT  grows  and 
develops  its  skills  and  IT  applications. 

The  first  two  goals  address  whether  IT  is  providing  the  right  products  and 
services  for  the  enterprise  and  individual  customers.  The  latter  two  goal 
areas  address  how  well  IT  is  performing  in  its  own  capability  to  deliver 
those  products  and  services.  The  strategic  and  customer  perspectives  are 
key  for  linking  to  mission  planning  requirements  in  the  Results  Act,  the 
Chief  Financial  Officers  Act,  the  Paperwork  Reduction  Act,  and  the  Clinger- 
Cohen  Act. 
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Managers  in  our  case  study  organizations  emphasized  that  "balance"  does 
not  necessarily  mean  "equality."  Use  of  a  balanced  approach  only  means  the 
consideration  of  several  goal  areas  and  the  development  of  objectives  and 
measures  in  each.  For  example,  Kodak  managers  liked  the  balanced 
scorecard  approach  because  it  was  a  multivariable  approach.  Before  using 
it,  the  IT  organization  was  very  cost  conscious  and  tended  to  judge 
investments  in  new  applications  or  skills  largely  from  a  cost  perspective. 

The  balanced  scorecard  examines  short-term  cost  goals  and  potential 
business  value  in  the  context  of  various  other  nonfinancial  operating 
parameters. 

2.  Focus  on  the  "Vital  Few"  Objectives  and  Measures 

Each  leading  organization  customizes  a  set  of  measures  appropriate  for  its 
organizational  goals  and,  for  IT,  how  IT  fits  into  the  enterprise's  strategic 
direction  and  mission  delivery  plans.  The  organizations  concentrate  their  IT 
performance  management  efforts  on  a  vital  few  objectives  and  measures 
within  the  goal  areas.  The  organizations  did  not  severely  limit  the  number 
of  measures  developed  at  the  beginning.  But,  over  time,  and  with 
experience,  the  organizations  became  more  focused  in  the  measures  they 
used.  However,  use  of  a  balanced  scorecard  approach  gets  rid  of  "safety 
net"  measures  which  organizations  often  collect  but  do  not  use  for  decision¬ 
making,  resource  allocation,  or  oversight  reporting  purposes. 

As  is  explained  in  the  sections  that  follow,  the  measure  examples  illustrate 
the  need  for  diversity.  Within  some  of  our  case  study  organizations,  similar 
measures  are  being  used,  but  the  measures  remain  under  development, 
requiring  more  refinement  and  documentation.  The  measures  presented 
here  do  not  represent  the  full  universe  of  what  an  organization  might  use. 
Also,  in  practice,  the  goal  and  objective  areas  may  be  more  specific  than 
those  presented  on  the  following  pages.  For  example,  one  of  our  goal  areas 
centers  on  the  customer  perspective.  One  objective  of  this  goal  area  is 
customer  satisfaction.  In  practice,  an  actual  customer  objective  statement 
might  be  stated  as  "This  fiscal  year,  at  least  98  percent  of  customers  will  be 
satisfied  with  IT  products,  services,  and  processes."  In  short,  the  following 
sections  discuss  a  general  categorization  of  IT  goals,  objectives,  and  sample 
measures. 


Balanced  Scorecard  Goal  Area  1: 

Achieving  the  Strategic  Needs  of  the  Enterprise 


IT  strategic  measures  are  designed  to  evaluate  the  aggregate  impact  of  IT 
investments  on  the  organization.  In  short,  these  measures  provide  insights 
into  impacts  made  by  the  organization's  entire  portfolio  of  IT  investments. 
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This  goal  area  focuses  on  ways  to  measure  how  IT  supports  the 
accomplishment  of  organizational  strategies.  The  strategic  perspective 
recognizes  that  in  successful  organizations,  all  components,  including  IT, 
must  align  with  enterprise  goals  and  directions. 

When  evaluating  the  impact  of  IT  in  terms  of  strategic  needs,  the  following 
questions  should  be  considered: 

•  How  well  integrated  are  our  IT  strategies  with  business  needs? 

•  How  well  is  the  overall  portfolio  of  IT  investments  being  managed? 

•  Is  IT  spending  in  line  with  expectations? 

•  Are  we  consistently  producing  cost-effective  results? 

•  Are  we  maximizing  the  business  value  and  cost  effectiveness  of  IT? 

IT  managers  and  staff  often  attempt  to  satisfy  individual  operational 
customers  without  a  check  against  enterprise  interests.  Having  this  goal 
area  prevents  targeting  IT  efforts  for  individual  operational  customers  which 
may  be  very  counter-productive  to  enterprise  IT  needs  and  expectations. 
Doing  so  is  difficult,  as  one  manager  said,  "It  has  been  a  cultural  plan  to 
look  at  [the  company]  as  a  whole  versus  maximizing  for  individual  business 
partners.  The  reward  and  incentive  systems  are  set  up  to  emphasize  that  all 
senior  managers  must  succeed  together." 

The  four  IT  strategic  enterprise  objectives  presented  in  figure  7  reflect 
several  key  objective  areas  of  the  organizations  we  studied.  These 
objectives  cover  enterprise  strategic  planning  and  goal  accomplishment, 
enterprise  management  of  the  portfolio  of  IT  applications,  IT  financial  and 
investment  performance,  and  use  of  IT  resources  across  the  enterprise. 

The  first  objective  in  this  goal  area  addresses  how  well  IT  plans  and  efforts 
reflect  enterprise  mission  goals.  This  objective  area  assumes  the  enterprise 
has  defined  its  mission  goals  and  can  make  the  clear  link  to  how  IT 
supports  those  goals.  The  sample  measures  capture  the  contribution  of  IT 
solutions  and  services,  compare  what  was  planned  for  IT  benefits  and  IT 
strategies  against  what  actually  happened,  and  compare  IT  strategies  and 
planning  and  enterprise  strategies  and  planning.  The  overall  measurement 
thrust  is  to  make  sure  that  enterprise  mission  goals  direct  IT  activities. 

The  second  objective,  portfolio  analysis  and  management,  is  a  growing 
concern  among  the  organizations  we  studied.  Leading  organizations  want  to 
make  sure  they  have  the  right  portfolio  of  IT  applications  either  planned  or 
in  place  that  will  enhance  business  or  mission  performance. 
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Figure  7:  Balanced  Scorecard-IT  Strategic  Measures 


Objectives 

Sample  Measures 

Enterprise 
mission  goals 

Percent  mission  improvements  (cost,  time,  quality) 
attributable  to  IT  solutions  and  services 

Percent  planned  IT  benefits  projected  v.  realized 

Portfolio  analysis 
and  management 

Percent  IT  portfolio  reviewed  and  disposed 

Percent  old  applications  retired 

Percent  applications  retirement  plan  achieved 

Percent  reusable  of  core  application  modules 

Percent  new  IT  investment  v.  total  spending 

Financial  and 

investment 

performance 

Percent  and  cost  of  services  provided  in-house  v.  industry 
standard 

IT  budget  as  a  percent  of  operational  budget  and  compared 
to  industry  average 

Net  present  value,  internal  rate  of  return,  return  on 
investment,  return  on  net  assets 

IT  resource  usage 

Percent  consolidated/shared  resources  across  units 

Percent  cross-unit  shared  databases  and  applications 

Percent  hardware/software  with  interoperability  capabilities 

Kodak  defines  an  application  portfolio  as  a  comprehensive  inventory  of  computer 
applications  that  were  developed  or  purchased  to  manage  an  organization's 
processes  and  information.  The  inventory  contains  detailed  data  relative  to  each 
application’s  size  and  characteristics,  effectiveness  in  meeting  business  needs, 
potential  for  growth,  and  development  and  maintenance  costs.  The  application 
portfolio  forms  the  foundation  for  an  overall  IT  investment  strategy. 

Xerox  has  defined  its  IT  inventory  in  a  similar  manner  and  made  IT  portfolio 
management  a  key  objective  area  as  part  of  its  overall  strategic  enterprise 
strategies.  As  described  in  an  earlier  case  study  describing  its  IM2000  strategy, 
Xerox  wanted  to  improve  information  management  spending,  deliver  IT 
infrastructure  renewal,  and  deliver  process-driven  IT  solutions  to  customers.  A  key 
part  of  the  overall  IT  strategy  was  to  evaluate  existing  IT  applications  and 
determine  how  they  supported,  if  at  all,  the  IM2000  strategy.  Xerox  ran  each  of 
its  existing  applications  through  a  rigorous  analysis  process,  categorizing  each  into 
one  of  nine  "disposition"  categories  ranging  from  stopping  those  of  low  usage  and 
value  to  keeping  others  as  corporatewide  applications. 
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For  Xerox,  the  IT  portfolio  strategy  helps  accomplish  several  performance  goals. 
The  strategy  reduces  unnecessary  operational  costs  and  increases  support 
productivity;  identifies  and  consolidates  similar  applications  and  eliminates  low 
value  applications;  identifies  and  retires  legacy  applications,  data,  and  infrastructure 
as  new  solutions  are  deployed;  and  identifies  consolidation  and  sharing 
opportunities.  The  principle  is  to  view  application  systems  as  corporate  assets.13 

The  third  objective  in  this  goal  area  examines  financial  and  investment 
performance.  While  the  two  objectives  above  cover  mission  goals  and  portfolio 
management,  this  objective  addresses  management  of  IT  costs  and  returns.  The 
sample  measures  capture  costs  in  major  IT  financial  categories  such  as  hardware 
and  software.  They  also  provide  information  on  the  balance  of  spending  between 
legacy  and  new  development  applications  and  between  in-house  and  outsourced 
operations.  Another  sample  measure  compares  the  IT  budget  to  the  enterprise 
operational  budget,  and  how  that  compares  to  industry  standards. 

Sample  measures  also  look  at  the  return  on  the  IT  investments,  offering  several 
different  methodologies  such  as  rate  of  return  and  net  present  value.  Much  of  this 
information  is  traditionally  benchmarked  with  other  IT  organizations  of  similar  size 
and  IT  penetration.  These  measures  are  tied  to  customer  and  enterprise  strategic 
perspectives  to  assess  if  scarce  resources  are  being  invested  wisely.  This  is  an 
especially  important  area  in  the  federal  government  with  the  emphasis  on  cost 
reduction  and  the  best  possible  use  of  existing  resources. 

Lastly,  IT  resource  usage  as  an  objective  targets  how  well  the  organization  can 
leverage  and  share  its  IT  resources  across  the  enterprise.  The  measures  evaluate 
factors  such  as  what  resources  can  be  shared,  what  has  been  consolidated,  and 
employee  access  to  computing  services.  From  a  strategic  perspective,  this 
objective  recognizes  the  need  for  shared,  enterprisewide  applications  and  the  use  of 
an  IT  infrastructure  and  architecture  for  the  entire  organization. 


13The  Office  of  Management  and  Budget  and  GAO  issued  a  guide,  Evaluating  Information 
Technology  Investments:  A  Practical  Guide,  in  November  1995  to  assist  federal  agency  and 
oversight  staff  in  evaluating  a  portfolio  of  information  technology  investments  in  a  similar  manner. 
The  approach  is  also  an  underlying  feature  of  GAO's  guide  Assessing  Risks  and  Returns:  A  Guide 
for  Evaluating  Federal  Agencies'  IT  Investment  Decision-making  (GAO/AIMD-IO.1.13,  February 
1997). 
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Balanced  Scorecard  Goal  Area  2: 

Satisfying  the  Needs  of  Individual  Customers 


IT  customer  measures  are  designed  to  measure  the  quality  and  cost  effectiveness  of 
IT  products  and  services.  When  evaluating  the  impact  of  IT  on  customer 
satisfaction,  the  following  questions  should  be  considered: 

•  How  well  are  business  unit  and  IT  staff  integrated  into  IT  systems 
development  and  acquisition  projects? 

•  Are  customers  satisfied  with  the  IT  products  and  services  being  delivered? 

•  Are  IT  resources  being  used  to  support  major  process  improvement 

efforts  requiring  information  management  strategies?  If  so,  are  the  IT  projects 
delivering  the  expected  share  of  process  improvement? 

The  purpose  of  the  second  goal  area  is  to  meet  the  needs  of  individual  operational 
customers.  The  three  objectives  shown  in  figure  8  capture  the  key  objective  areas 
we  found  in  our  research. 

Figure  8:  Balanced  Scorecard— IT  Customer  Measures 


Objectives 

Sample  Measures 

Customer 
partnership  and 
involvement 

Percent  projects  using  integrated  project  teams 

Percent  joint  IT  customer/supplier  service-level  agreements 

Customer 

satisfaction 

Percent  customers  satisfied  with  IT  product  delivery 

Percent  customers  satisfied  with  IT  problem  resolution 

Percent  customers  satisfied  with  IT  maintenance  and  support 

Percent  customers  satisfied  with  IT  training 

Percent  products  launched  on  time 

Percent  service-level  agreements  met 

Business  process 
support 

Percent  IT  solutions  supporting  process  improvement  projects 

Percent  users  covered  by  training  to  use  new  IT  solutions 

Percent  new  users  able  to  use  applications  unaided  after  initial 
training 

Two  of  the  objective  areas,  customer  satisfaction  and  business  process 
support,  address  direct  IT  support.  Customers  were  especially  interested  in 
time,  cost,  quality,  overall  customer  satisfaction,  and  business  process 
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support.  One  official  we  talked  to  said,  "[Our  IT  organization]  looks  at  the 
business  process  characteristics  of  our  customers.  IT  personnel  ask  are 
there  better  ways  to  support  product  innovation  and  development?  How 
does  IT  support  that?  The  question  is  the  effectiveness  of  IT  in  supporting 
business  processes~not  cranking  out  function  points."14 

The  first  objective  area,  customer  partnership  and  involvement,  stresses  a 
mutual  partnership  between  the  IT  organization  and  customers  in  developing 
the  best  possible  IT  products  and  services.  The  sample  measures  examine  a 
variety  of  areas,  ranging  from  cooperation  and  joint  development  to 
involvement  in  project  management. 

Customer  satisfaction  measures  assess  how  well  customers  are  satisfied 
with  many  IT  activities.  Sample  measures  also  cover  the  accomplishment  of 
system  design  requirements,  complaints,  problem  resolution,  error  and 
defect  rates,  timeliness,  and  service-level  agreement  accomplishments. 

The  business  process  support  objective  area  emphasizes  the  importance  of 
business  process  improvement  as  organizations  streamline  and  reengineer. 
Business  process  improvement  is  a  central  objective  area  for  many  of  the 
organizations  we  studied.  The  sample  measures  capture  how  well  IT 
supports  business  process  improvement  plans  and  process  analysis.  They 
also  examine  the  adaptability  of  IT  solutions,  training  for  new  IT  solutions 
and  the  effectiveness  of  the  training,  and  costs  in  moving  applications  to 
new  hardware. 


Balanced  Scorecard  Goal  Area  3: 

Addressing  IT  Internal  Business  Performance 


Internal  IT  business  measures  are  designed  to  evaluate  the  operational 
effectiveness  and  efficiency  of  the  IT  organization  itself.  The  ability  of  the 
IT  shop  to  deliver  quality  products  and  services  could  have  a  direct  impact 
on  decisions  to  outsource  IT  functions.  When  evaluating  internal  IT 
business  functions,  the  following  questions  should  be  considered: 

•  Are  quality  products  delivered  within  general  industry  standards? 

•  Are  quality  products  being  delivered  using  accepted  methods  and  tools? 

•  Is  our  infrastructure  providing  reliable  support  for  business  needs? 


14A  function  point  measures  an  IT  application  in  terms  of  the  amount  of  functionality  it  provides 
users.  Function  points  count  the  information  components  of  an  application,  such  as  external  inputs 
and  outputs  and  external  interfaces. 
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•  Is  the  enterprise  architecture  being  maintained  and  sustained? 

One  manager  we  interviewed  said,  "There  are  two  dimensions  of  [IT] 
performance.  One  is  the  dominant  or  visible  component-the  use  of  IT  in 
the  context  of  [customer]  business  processes.  The  other  is  transparent-the 
functional  excellence  of  IT."  The  first  two  goal  areas  stress  the  use  of  IT  as 
it  supports  enterprise  and  operational  customers.  On  a  day-to-day  basis,  it  is 
the  functional  excellence  of  IT  internal  business  processes  which  delivers 
that  support.  Figure  9  shows  four  objective  areas  and  sample  measures  we 
synthesized  from  our  case  study  organizations  and  the  general  IT  literature. 


Figure  9:  Balanced  Scorecard— IT  Internal  Business  Measures 


Objectives 

Sample  Measures 

Applications 
development  and 
maintenance 

Number  of  function  points  delivered  per  labor  hour 

Number  of  defects  per  100  function  points  at  user  acceptance 

Number  of  critical  defects  per  100  function  points  in 
production 

Percent  decrease  in  application  software  failures,  problems 

Mean  time  to  resolve  critical  defects 

Cycle  time  for  development 

Project  performance 

Percent  projects  on  time,  on  budget 

Percent  projects  meeting  functionality  requirements 

Percent  projects  using  standard  methodology  for  systems 
analysis  and  design 

Infrastructure 

availability 

Percent  computer  availability 

Percent  communications  availability 

Percent  applications  availability 

On-line  system  availability 

Enterprise 

architecture 

standards 

compliance 

Number  of  variations  from  standards  detected  by  review  and 
audit  per  year 

Percent  increase  in  systems  using  architecture 

Percent  staff  trained  in  relevant  standards 

IT  managers  and  staff,  along  with  enterprise  senior  management,  decide 
which  of  the  many  IT  processes  truly  must  excel  for  meeting  customer  and 
enterprise  goals  in  the  short  and  long  term.  For  example,  is  the  IT  process 
for  identifying  the  right  technology  for  customer  applications  the  best  it  can 
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be?  IT  managers  and  staff  set  specific  goals  for  improvement  of  internal  IT 
business  processes. 

The  first  objective  covers  IT's  performance  in  developing  and  maintaining 
applications.  The  sample  measures  include  dollars  expended  per  function 
point,  average  application  development  cycle  time,  and  cost.  The  second 
objective  area  examines  the  performance  in  delivering  projects,  capturing 
traditional  measurements  on  project  time,  budget,  functionality,  and  use  of 
widely  accepted  methods  and  tools.  Measures  also  capture  backlogs  in  both 
development  and  enhancement  or  maintenance  of  applications. 

The  third  objective  area  addresses  IT  infrastructure  availability  in  a  variety 
of  areas,  as  well  as  response  time  and  transactions.  Many  of  the 
organizations  we  studied  stressed  the  importance  of  infrastructure 
availability,  an  area  totally  transparent  to  the  customer  until  something  goes 
wrong.  These  measures  keep  managers  on  top  of  infrastructure 
performance  where  there  is  little  tolerance  for  down  time.  The  last 
objective  area  covers  architectural  standards.10  The  measures  assess  how 
well  IT  is  meeting  set  standards,  most  often  developed  for  interconnectivity 
and  interoperability  and  efficient  IT  support. 

Many  of  the  traditional  IT  measures  fall  into  the  internal  business 
performance  goal  area,  often  focusing  on  the  efficiency  of  computing  and 
communications  hardware  and  software.  The  measures  in  this  goal  area 
frequently  are  used  for  individual  manager  and  staff  IT  accountability,  as 
described  in  a  later  practice. 

Some  of  the  organizations  we  studied  were  using  the  Software  Engineering 
Institute's  five-level  capability  maturity  model  to  guide  their  IT  process 
improvement  efforts.  The  objective  areas  and  measures  are,  in  contrast  to 
some  of  the  other  balanced  scorecard  areas,  highly  integrated.  For  example, 
project  performance  relies  on  effective  applications  development  and 
maintenance. 


15IT  architectures  explicitly  define  common  standards  and  rules  for  both  data  technology,  as  well  as 
mapping  key  processes  and  information  flows.  A  complete  IT  architecture  should  consist  of  both 
logical  and  technical  components.  The  logical  architecture  provides  the  high-level  description  of  the 
organization's  mission,  functional  requirements,  information  requirements,  systems  components,  and 
information  flows  among  the  components.  The  technical  architecture  defines  the  specific  IT 
standards  and  rules  that  will  be  used  to  implement  the  logical  architecture. 
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Balanced  Scorecard  Goal  Area  4: 
Addressing  Innovation  and  Learning 


Innovation  and  learning  measures  evaluate  the  IT  organization's  skill  levels  and 
capacity  to  consistently  deliver  quality  results.  This  goal  area  recognizes  that 
without  the  right  people  with  the  right  skills  using  the  right  methodologies,  IT 
performance  will  surely  suffer.  Measures  in  this  goal  area  should  be  used  to 
answer  the  following  questions: 

•  Do  we  have  the  right  skills  and  qualified  staff  to  ensure  quality  results? 

•  Are  we  tracking  the  development  of  new  technology  important  to  our 
business/mission  needs? 

•  Are  we  using  recognized  approaches  and  methods  for  building  and  managing 
IT  projects? 

•  Are  we  providing  our  staff  the  proper  tools,  training,  and  incentives  to  perform 
their  tasks? 

The  four  objective  areas  shown  in  figure  10  include  workforce  competency  and 
development,  advanced  technology  use,  methodology  currency,  and  employee 
satisfaction  and  retention. 

Figure  10:  Balanced  Scorecard-IT  Innovation  and  Learning 
Measures 


Objectives 

Sample  Measures 

Workforce 
competency  and 
development 

Percent  staff  trained  in  use  of  new  technologies  and 
techniques 

Percent  staff  professionally  certified 

Percent  IT  management  staff  trained  in  management  skills 

Percent  IT  budget  devoted  to  training  and  staff  development 

Advanced  technology 

use 

Percent  employees  skilled  in  advanced  technology  applications 

Number  of  dollars  available  to  support  advanced  technology 
skill  development 

Methodology 

currency 

Currency  of  application  development  methods  used 

Percent  employees  skilled  in  advanced  application 
development  methods 

Percent  projects  developed  using  recognized  methods  and 
tools 

Employee 
satisfaction  and 
retention 

Percent  employee  satisfaction  with  the  capability  of  the 
existing  technical  and  operating  environment  to  support 
mission 

Percent  employee  turnover  by  function 
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This  goal  area  develops  the  continuous  improvement  aspect  of  IT  activities. 

It  speaks  to  capabilities  of  bringing  new  technologies  to  bear  on  customer 
problems,  practicing  the  best  methodologies,  and  retaining  and  developing 
the  best  employees.  The  first  objective  area  stresses  the  importance  of 
having  a  capable  and  competent  workforce.  In  particular,  the  organizations 
we  studied  were  very  concerned  with  workforce  competence  and 
development.  Key  measures  included  training  hours  and  skill  development. 
Most  were  transitioning  from  core  competencies  in  operations  and 
maintenance  to  business  process  improvement  and  reengineering,  new 
business  solutions,  and  technical  direction  of  applications  development  done 
by  others. 

The  second  and  third  objectives,  advanced  technology  use  and  methodology 
currency,  speak  to  the  ability  to  recognize  and  deploy  advanced  technologies 
and  methodologies  in  doing  IT's  work.  The  last  objective,  employee 
satisfaction  and  retention,  measures  how  well  employees  themselves  are 
satisfied  with  the  quality  of  their  work  environment  and  general  IT  strategies 
and  accomplishments. 


How  to  Get  Started 

To  begin  developing  a  balanced  scorecard  approach  for  IT, 

organizations  should 

•  get  agreement  among  business  and  IT  management  on  the 
approach  that  will  be  used  for  developing  IT-related 
performance  indicators  and  measures, 

•  using  the  agreed  upon  approach,  define  and  develop  the  key 
goal  areas  and  objectives  for  the  IT  organization, 

•  develop  a  full  set  of  measures  in  one  or  two  priority  IT  goal 
areas,  then  expand  out  to  other  goal  areas,  and 

•  test  the  performance  measurement  system  and  make  revisions 
based  upon  initial  lessons  learned. 
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Practice  Area  3: 
Target  Measures, 
Results,  and 
Accountability  at 
Decision-making  Tiers 


ALIGN 


REINFORCE 


Practice  Area  Characteristics 

1.  Track  IT  measures  and  appropriate  reports  to  each  decision¬ 
making  level 

2.  Align  measures  from  the  bottom  to  the  top 

3.  Directly  link  tiered  measures  to  the  balanced  scorecard 

4.  Align  individual  accountability  to  IT  scorecard  goals 


Practice  Area  Overview 

Organizations  in  our  study  targeted  measures  and  performance  reports  at 
specific  decision-making  levels  or  tiers.  These  tiers  cover  the  enterprise 
(agency)  level,  senior  to  mid-level  management  (program)  level,  and 
specific  operations  or  project  level.  This  approach  offers  several 
advantages,  including  (1)  enhanced  communication  and  understanding  of 
performance  measurement  throughout  the  organization,  (2)  systematic 
links  among  measures  and  enterprise,  program,  project,  and  individual 
objectives,  and  (3)  alignment  of  measures  with  mission  results.  IT 
performance  information  should  drive  management  actions  and  decisions 
that  support  the  attainment  of  organizational  goals. 

Practice  Area  Characteristics 

1.  Track  IT  Measures  and  Reports  to  Decision-making  Levels 

As  shown  in  figure  11,  performance  measures  and  reports  at  each  tier 
have  specific  purposes.  At  the  enterprise  tier,  IT  performance  and 
measures  focus  on  mission  results,  or  how  well  IT  is  meeting  its  purpose 
in  supporting  enterprisewide  goals  and  objectives.  Information  on  final 
and  intermediate  outcomes  of  programs  facilitated  by  IT  projects  and 
investments  would  be  shown.  A  summary  report  may  be  prepared  for 
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external  reporting  to  stakeholders  and  the  general  public.  Reports  may 
be  prepared  on  an  annual  or  quarterly  basis  and  highlight  IT  policy- 
oriented  information  showing  areas  of  progress,  problems,  and 
contextual  or  explanatory  information  to  supplement  the  performance 
data. 


Figure  11:  Performance  Measurement  Tiers 


Measurement  focus  and 
timing 


Cyclical  (e.g.,  annual  or  quarterly) 
information  that  focuses  on  mission 
results  used  for  policy  decisions  and 
strategies 


Periodic  (e.g.,  quarterly  or  monthly) 
information  that  focuses  on  unit  results 
associated  with  management  and 
operational  improvements 


Immediate  (e.g.,  daily  or  weekly) 
information  that  focuses  on  activity 
and  task  level  data  used  to  make 
tactical  decisions  and  execute 
management  decisions 


At  the  IT  program  or  support  unit  level,  senior  to  mid-level  managers 
want  to  know  how  specific  units  or  processes  are  performing. 
Measurement  most  often  covers  specific  IT  business  processes  such  as 
applications  development  or  lines  of  business  (or  core  business  areas). 

At  this  tier,  more  detailed  performance  information  is  used  for 
management  and  the  improvement  of  operations  and  integrating  activities 
across  IT  processes  or  programs. 

In  the  third  tier,  or  bottom  level,  the  measurement  emphasis  is  generally 
at  the  project  level  and  individual  systems  performance.  Highly  detailed 
tactical  and  execution  information  supports  immediate  and  day  to  day 
decision-making  on  funding,  contract  development  and  monitoring, 
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project  priorities,  and  possible  adjustments  in  program  operating 
procedures.  Here  the  emphasis  is  on  input  and  output  measures. 

Tiering  in  this  manner  can  help  assign  performance  accountability  and 
determine  where  IT  measurement  ownership  lies.  The  organizations 
decide  where  pieces  of  IT  performance  accountability  rest,  such  as  user 
or  operational  departments,  central  IT  organizations,  and/or  department 
or  operational  IT  units.  In  decentralized  organizations,  many  people  are 
involved  in  the  IT  processes  that  deliver  products  and  services. 


Case  Study  3: 

Using  Performance  Measures  for  Different  Management  Tiers 

Xerox  information  management  operational  measures  are  targeted  at  three  distinct 
organizational  tiers.  As  shown  in  the  following  figure,  Tier  1  consists  of  corporatewide 
metrics,  targeted  for  senior  managers  and  executives.  Tier  2  focuses  on  measures  used  by 
specific  organizations,  divisions,  and  core  business  areas.  These  measures  assess  what  is 
happening  below  the  enterprise  level  and  actually  roll  up  Tier  3  metrics  collected  at  the 
project  level. 


Case  Study  3: 

Xerox  Tiered  Information  Management  Matrix 


Tier  1:  Global  Information  Management 

» IT  spending  as  %  of  revenues  reduced  (quarterly) 

>  Total  IT  spending  as  %  of  IT  investment  funding  (quarterly) 

>  Customer  satisfaction  scores  (annually) 


f 


Tier  2:  Organizations,  Divisions,  Core  Areas 

►  %  projects  on  time  (actual  versus  planned) 

►  costs/benefits  for  IT  investments  (projects  and  overall) 

►  overall  IT  spending  profiles  (all  IT  investments) 


Tier  3:  Project  Level  (contractor  performance 

■  $  per  function  point  delivered  •  %  projects  on  time 
%  software  delivery  re-use  •  productivity  (function  point 

■  function  point  defects  per  month) 


Consolidate 
and  analyze 

Identify 

trends 

Communicate 

results 

Benchmark 


GAO/AIMD-98-89  IT  Performance  Measurement  Guide 


Page  47 


Case  Study  4: 

Decision-making  Levels  Use  Different  Types  of  Measures 

American  Express'  Technologies  group  is  also  developing  a  tiered  measurement  process  for 
decision-making  at  all  management  levels.  Tier  1  measures  consist  of  executive 
information  that  represents  the  Technologies  group's  overall  effectiveness  against  such  goals 
as  (1)  achieving  world  class  time-to-market ,  (2)  developing  new  business >  and  (3) 
enabling  business  partners .  Specific  measures  include  development  cycle  time)  time 
requirements  for  ongoing  operations ,  cost  of  ongoing  operations ,  cost  of  quality ,  leading 
quality  indicators ,  and  elimination  of  root  causes  of  failures. 

Tier  2  measures  consist  of  management  information  for  senior  and  mid-level  managers 
and  have  direct  links  to  tier  1  measures .  To  illustrate ,  tier  2  measures  supporting  the  tier 
1  measure  of  development  cycle  time  include  " elapsed  time  per  function  point "  and  " effort 
hours  per  function  point . " 

Tier  3  measures  are  operational  information  used  for  project  development ,  operations ,  and 
project  leaders.  Tier  3  information  links  directly  to  tier  2  and  forms  the  basis  for 
management  decisions ,  root  cause  analysis ,  and  continuous  process  improvement 
evaluations.  Specific  performance  measures  are  used  to  evaluate  individual  projects  and 
applications. 


2.  Align  Measures  From  the  Bottom  to  the  Top  of  the 
Organization 

A  key  performance  feature  found  in  the  organizations  we  studied  is  the 
notion  of  aligning-but  not  necessarily  "rolling-up"-measures  from  the 
bottom  to  the  top  of  the  organization.  IT  measures  used  at  the  lowest 
tier--the  specific  operations  or  project  level--must  align  upwards  with  the 
subsequent  tiers.  In  other  words,  the  IT  input  and  output  information 
collected  for  tactical  and  execution  management  must  directly  relate  to 
unit  results  needs  and  then  upwards  to  the  enterprise  level.  This 
alignment  helps  to  ensure  that  performance  measures  and  information  at 
the  lowest  level  directly  support  policy  and  mission  decisions  and 
strategies. 

Only  rarely  will  an  organization  have  a  single  performance  measure 
appropriate  for  all  three  levels,  or,  in  other  words,  those  that  can  "roll¬ 
up"  to  the  top  of  the  pyramid.  A  temptation  in  performance  measures  is 
to  layer  measures  from  lower  levels  on  top  of  each  other  and  pass  the 
information  along  to  higher-level  officials.  This  approach  may  provide  an 
overload  of  information  not  easily  understandable  or  digestible  by  top 
executives  or  even  the  public.  It  also  creates  the  potential  of  hiding 
embarrassing  details  in  a  mountain  of  data,  or  can  promote  the  self- 
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selection  of  favorable  data.  A  few  of  the  bottom  tier  measures  may  "roll 
up"  into  strategic  measures  of  interest  at  the  other  two  tiers.  However, 
the  type  and  formatting  of  IT  information  and  measures  and  timing  of 
performance  reporting  appropriate  at  one  tier  may  not  be  appropriate  for 
others.  Use  of  all  three  tiers  gives  the  organization  a  comprehensive 
picture  of  the  value  of  IT  and  if  the  individual  IT  products,  services,  and 
processes  were  worth  the  investment. 

3.  Directly  Link  Tiered  Measures  to  a  Balanced  Scorecard 

Once  the  IT  organization  has  agreement  on  balanced  scorecard  goals  and 
objectives,  then  it  would  develop  limited  tiered  measures  to  address 
specific  operational  or  project  measures,  program  and  support  unit 
measures,  and  enterprise-level  measures.  The  tiering  of  measures  across 
the  balanced  scorecard  area  facilitates  the  use  of  measures  for  decision¬ 
making  and  performance  improvement  action.  Otherwise,  if  measures 
are  not  used,  they  lose  their  decision-making  impact,  which  results  in 
less  effort  to  collect  and  use  them  and,  in  turn,  leads  to  less  decision¬ 
making  impact. 

One  tier  is  not  more  important  than  the  other  two.  As  one  manager 
noted,  an  operational  customer  will  never  ask  for  reports  containing 
performance  data  found  in  the  bottom  tiers,  such  as  mainframe 
availability  or  application  availability.  But  this  operational  data  provides 
vital  information  about  how  well  IT  is  supporting  program  operations  and 
is  indirectly  linked  to  customer  satisfaction  measures. 

The  use  of  performance  measurement  tiers  is  not  a  novel  concept,  but 
reflects  a  change  in  how  measurement  information  is  used  by  an 
organization.  Traditional  IT  performance  measures  are  used  to  examine 
lines  of  code  generated,  number  of  reports  issued,  data  center  downtime, 
transactions,  and  the  number  of  projects  on  time.  These  can  be 
considered  bottom  tier  measures.  More  recently,  management  emphasis 
has  shifted  towards  performance-oriented  customer  requirements, 
productivity  and  quality  improvements,  selection  of  strategic  projects, 
and  weighing  value  of  delivered  systems. 

Using  some  specific  objectives  under  the  balanced  scorecard  goal  areas 
discussed  in  Practice  2,  figure  12  provides  some  hypothetical  examples  of 
IT  measures  that  might  be  used  at  different  organizational  tiers.  A 
combination  of  input,  output,  and  outcome  measures  are  sprinkled 
throughout  the  tiers  to  accommodate  different  management  information 
decision-making  and  reporting  needs. 
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Figure  12:  A  Tiered  Performance  Scorecard  Example 


r 

Scorecard 

Tiered  Measures 

a 

Goals 

Objectives 

Specific 
Operational 
or  Project 
Measures 

Program/ 
Support  Unit 
Measures 

Enterprise 

Level 

Measures 

Strategic 

enterprise 

perspective 

Portfolio 
analysis  and 
management 

%  reusability  of 
core  modules 

%  compliance  to 
approved  IT 
solution 
deployment 

%  IT  portfolio 
reviewed  and 
disposed 

Customer 

perspective 

Customer 

satisfaction 

#  complaints  to 
help  desk 

%  customers 
satisfied  with  IT 
application 
design 

%  planned 
system  design 
requirements 
realized  vs. 
deliverables 

Internal 

business 

perspective 

Applications 

development 

and 

maintenance 

#  defects  per 

1 00  function 
points  at  user 
acceptance 

%  decrease  in 
application 
software  failures 

%  projects  on 
time,  on  budget 

Innovation  & 
learning 
perspective 

Workforce 
competency  & 
development 

#  staff  by  skill 
area 

%  staff 

professionally 

certified 

%  IT  budget 
devoted  to 
training  and  staff 
development 

4.  Align  Individual  Accountability  to  IT  Goals 

The  last  question  that  ties  together  the  IT  balanced  scorecard  and  the  IT 
results  chain  is  "Who  is  accountable  for  results  and  how  are  they  held 
accountable?"  The  leading  organizations  have  learned  that  managing 
performance  well  depends  on  making  the  connection  between  program 
and  IT  purpose,  goals,  and  objectives,  and  responsible  teams  and 
individual  staff. 

Alignment  begins  when  organizational  and  program  goals  and  objectives 
are  translated  into  action  plans  for  the  improvement  of  IT  products, 
services,  systems,  processes,  and  practices.  Just  as  measure 
development  involves  many  staff,  IT  performance  plans  and  targets  must 
be  communicated  to  all  levels  of  the  organization.  Smaller  components 
develop  goals  and  objectives  that  support  the  higher  level.  In  turn, 
individual  accountability  is  based  on  the  actions  that  each  individual  can 
take  to  contribute  to  the  organization's  goals.  For  example,  the  system 
of  accountability  must  recognize  that  the  final  outcomes  of  IT  support 
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activities  are  usually  intermediate  outcomes  for  the  higher  level  program 
goals  and  targets. 

Actual  levels  of  performance  should  be  tracked  against  targets,  and  both 
IT  and  program  managers  should  be  held  accountable  for  the  results. 
Measure  and  results  reviews  should  report  on  the  actual  outcomes  and 
individual  performance  appraisals  should  link  IT  performance  to  merit 
pay,  bonuses,  promotional  opportunities,  and  other  incentives. 


Case  Study  5: 

Aligning  Individual  Performance  With  Organizational  Goals 

Texas  Instruments  Information  Systems  and  Services  (IS&S)  business  excellence  improvement  process 
has  many  events  involving  measures  and  results.  Operational  assessments  include  quarterly  metrics 
and  project  reviews.  Information  sharing  events  include  a  presentation  of  annual  results  and  goals  in 
January  and  quarterly  communication  meetings  and  results  reports.  The  assessments  focus  on 
sharing  lessons  learned,  identifying  opportunities  to  transfer  best  practices,  uncovering  potential  risks, 
and  assigning  corrective  action.  IS&S  holds  quarterly  communication  meetings  open  to  all  IS&S  staff 
to  discuss  current  topics  and  answer  general  questions.  IS&S  leaders  use  quarterly  department 
meetings  to  further  disseminate  information  and  plans.  Several  electronic  information  sharing 
sources  are  available  to  IS&S  staff. 

IS&S  uses  a  performance  evaluation  and  development  process  to  allow  IS&S  personnel  the 
opportunity  to  detail  their  previous  year  s  accomplishments  and  identify  short  and  long-term  job  and 
career  goals.  The  employee  uses  the  process  to  align  his/her  goals  to  those  of  the  organization.  The 
process  is  key  to  promotions,  bonuses,  and  selection  for  a  technical  ladder--a  designation  recognizing 
technical  staff  for  outstanding  technical  contributions  and  quality  leadership.  IS&S  also  uses 
recognition  actions  to  support  quality  and  performance  improvement.  For  example,  recognition 
display  boards  are  located  in  highly  visible  areas.  These  display  individual  and  team  pictures  of 
IS&S  personnel  honored  for  technical  and  quality  contributions,  as  well  as  for  involvement  in  team 
problem  solving  activities  and  community  projects. 

Each  year,  IS&S  leadership  sends  a  memorandum  to  Texas  Instruments'  top  management  on  policy 
deployment  and  key  performance  measures.  Policy  deployment  is  Texas  Instruments'  term  for 
aligning  the  individual  to  overall  corporate  business  objectives  and  for  continuous  or  dramatic 
improvement  of  key  performance  measures. 

The  memorandum's  performance  areas  match  the  Texas  Instruments  strategic  directions  described  in 
an  earlier  chapter--customer  satisfaction,  continuous  improvement,  people  involvement,  and  cycle  time 
improvement.  Texas  Instruments  chose  cycle  time  as  a  measure  to  drive  dramatic  improvement  in  a 
specific  IT  process  (solutions  provisioning).  Improving  this  one  process  as  a  starting  point  addresses 
IT  structural  problems  that  affect  cycle  time.  The  IS&S  leadership  team  has  a  business  excellence 
improvement  plan.  This  plan  leads  to  action  plans  for  each  IS&S  division  supporting  the  strategic 
business  units  and  other  key  organizational  units,  to  team  action  plans  to  individual  contributions. 


Although  this  approach  is  controversial  in  the  pubic  sector,  leading 
private  sector  organizations  have  found  that  failure  to  translate 
organizational  strategy  into  program,  team,  and  individual  IT  goals  can 
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result  in  a  focus  on  short-term  and  tactical  issues  rather  than  the 
accomplishment  of  strategic  goals. 


How  to  Get  Started 

To  begin  targeting  measures,  results,  and  accountability  at 

different  decision-making  levels,  organizations  should 

•  identify  enterprise  tier  IT  performance  information 
requirements, 

•  use  the  enterprise  tier  information  to  develop  measures  across 
one  balanced  scorecard  area, 

•  begin  adjusting  management  performance  review  processes  at 
the  unit,  team,  and  individual  level  to  reflect  the  tiered 
approach,  and 

•  test  the  performance  measurement  system  and  make  revisions 
based  on  initial  implementation  efforts. 
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Practice  Area  4: 

Build  a  Comprehensive 
Measurement,  Data 
Collection, 

and  Analysis  Capability 


ALIGN 


IMPLEMENT 


CONSTRUCT 


REINFORCE 


Practice  Area  Characteristics 

1.  Use  data  collection  tools 

2.  Develop  and  use  baseline  and  benchmark  information 

3.  Assess  maturity  and  develop  complete  performance 
definitions 

4.  Utilize  concise,  understandable  performance  reporting 

5.  Conduct  measure  reviews  and  audits 


Practice  Area  Overview 


Building  a  balanced  scorecard  and  tiered  measures  is  only  one  step  in 
designing  an  effective  IT  performance  management  system.  In  the 
organizations  we  studied,  management  paid  careful  attention  to  the  "back 
end"  of  performance  management:  data  collection  and  analysis.  As  one 
manager  explained,  "You  need  a  lot  of  analysis  to  figure  out  the  key 
drivers  for  performance,  like  what  will  it  take  to  raise  customer 
satisfaction  one  or  more  percentage  points.  And  is  that  goal  good  for 
other  measures  such  as  cost  and  time?  What  is  the  gain  for  the  effort  we 
must  put  in  and  is  it  worth  it?" 

Most  organizations  began  by  benchmarking  existing  performance  against 
different  IT  units  within  the  organization,  external  IT  organizations  in 
other  businesses,  or  industry  benchmarks  published  by  research 
organizations.  Next,  performance  definitions  were  agreed  upon  and 
existing  data  used  to  "baseline"  existing  performance  and  identify 
information  gaps  that  needed  to  be  addressed  with  additional  data 
collection  and  analysis. 

Performance  data  are  needed  at  all  tier  levels.  Even  so,  the  collection 
and  reporting  of  this  information  should  not  impose  an  unnecessary 
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burden  on  management  and  staff.  Data  collection  should  utilize  efficient 
manual  or  automated  methods.  The  organizations  we  studied  developed 
a  clear  rationale  for  new  and  continued  data  collection  and  specifications 
for  accuracy,  reliability,  timeliness,  and  use  before  setting  out 
measurement  reporting  requirements.  Most  organizations  we  examined 
designed  efficient  and  effective  ways  to  present  the  performance 
information  to  management  so  that  it  could  facilitate  better  decision¬ 
making.  Finally,  the  organizations  regularly  conducted  reviews  of  their 
performance  measurement  systems  and  revised  or  updated  measures  in 
accordance  with  management  feedback  or  changing  business  needs. 

Practice  Area  Characteristics 


1.  Use  Data  Collection  Tools 

For  each  data  collection  requirement-whether  qualitative  or  quantitative- 
most  of  our  case  study  organizations  developed  manual  and  automated 
tools  to  reduce  the  burden  of  collecting  IT  performance  information. 
These  included  personal  observation,  formal  performance  measure 
reports,  customer  satisfaction  surveys  and  interview  questions,  reviews 
of  records  and  documents,  and  automated  hardware  and  software 
productivity  data  collection  tools. 


Case  Study  6: 

Collecting  IT  Performance  Information  Using  Multiple  Techniques 
Across  Different  Performance  Dimensions 

The  Immigration  and  Naturalization  Service's  Office  of  Information  Resources  Management 
(OIRM)  collects  information  on  both  its  Information  Technology  Partnership  (ITP)  contract , 
the  primary  IT  support  services  contract ,  and  for  program-level  performance  regarding 
mission  outcomes.  As  shown  in  the  following  figure ,  the  ITP  contract  tasks  measure  within 
budget ,  on-time ,  customer  satisfaction,  and  technical  ( such  as  quality  of  project 
completeness,  use  of  resources,  and  completeness  and  quality  of  documentation)  performance . 
The  program-level  performance  measures  cover  cost,  efficiencies,  and  quality  in  the 
improvements  realized. 

The  program  level  covers  all  major  operational  areas ,  such  as  the  Border  Patrol.  INS  has 
specifically  designated  tools  for  its  data  collection  under  the  Information  Technology 
Partnership  mentioned  earlier.  These  tools  include  a  performance  measures  report,  a 
customer  satisfaction  survey,  and  a  technical  performance  evaluation  form.  For  mission- 
critical  tasks,  quantitative  methods,  ratios,  and  performance  reports  are  the  tools  that  are 
used  for  data  collection  and  analysis. 
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Case  Study  6: 

INS  ITP  Performance  Measurement  Plan 


All  ITP  Contract  Tasks 


Most  of  the  organizations  we  studied  feature  customer  surveys, 
interviews,  and  focus  groups  as  important  sources  of  performance 
information.  These  data  collection  exercises  are  well-designed,  tailored 
to  IT  performance  measurement  ownership,  and  fit  into  management 
processes  to  take  action  on  the  results.  For  example,  Motorola  uses  a 
companywide  survey  that  asks  for  ratings  ranging  from  "superior"  to 
"poor"  on  such  issues  as 

•  availability  of  Information  Systems  (IS)  personnel, 

•  responsiveness  of  IS  personnel, 

•  effectiveness  of  IS  in  project  management  communication, 

•  reliability  of  IS  in  meeting  commitments, 
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•  IS  cycle  time  for  completing  projects, 

•  quality  of  IS  work, 

•  rate  of  improvement  during  the  past  year, 

•  alignment  of  IS  resources  with  business  needs, 

•  overall  credibility  of  information  systems  support,  and 

•  IS  performance  with  respect  to  total  customer  satisfaction. 

In  all  cases,  the  organizations  started  with  what  they  considered  a 
primitive  data  collection  strategy,  beginning  with  data  definitions  and 
working  with  customers  and  IT  staff  to  gain  understanding  and 
agreement.  In  the  early  stages,  some  of  the  data  collection  may  all  be 
manual,  later  supplemented  or  replaced  by  automated  systems.  All  of 
the  organizations  cautioned  that  IT  should  never  wait  for  automated  data 
collection  systems,  but  start  small  and  with  measures  meaningful  to 
enterprise  and  operational  customers,  such  as  IT  cycle  time  and 
customer  satisfaction. 
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Case  Study  7: 

Using  Multiple  Data  Collection  Mechanisms  and  Ensuring  System 
Integrity 

Texas  Instruments'  Information  Systems  &  Services  ( IS&S )  gathers ,  prioritizes,  and 
communicates  customer  and  supplier  concerns  through  a  series  of  teaming  activities .  IS&S 
conducts  annual  strategic  intent  conferences  and  objective  reviews  with  top-level  customer 
management.  Several  times  a  year  senior  customer  managers  on  an  Internal  Information 
Systems  Leadership  Team  meet  with  IS&S  managers  to  offer  strategic  planning  and 
guidance.  Other  activities  include  ongoing  executive  one-on-one  interviews  and 
management-level  and  project-level  customer  surveys,  employee  development,  reengineering 
efforts,  customer  focus  workshops,  steering  teams,  quality  improvement  teams,  leadership 
teams,  and  quality  steering  teams.  Senior  IS&S  management  uses  these  opportunities  to 
confer  with  the  customer  on  business  priorities,  critical  success  factors,  and  operational 
plans.  A  service-focused  survey  is  sent  to  end  users  of  IS&S  products  and  services.  Each 
Texas  Instruments  organization  has  a  metrics  coordinator  who  handles  the  manual  and 
automated  databases. 

Much  of  Texas  Instruments'  IS&S  performance  information  has  been  tracked  for  at  least  5 
years ,  with  the  majority  of  the  information  available  online.  A  combination  of  software 
error  checking,  password  control,  and  independent  audits  ensures  the  reliability, 
accessibility,  and  integrity  of  the  performance  databases.  IS&S  promotes  system 
consistency  and  maximum  data  sharing  through  standardized  data  formats  and  system 
interfaces.  IS&S  uses  an  electronic  data  interchange  network  to  communicate  electronically 
with  suppliers  and  customers.  An  online  complaint  system  assists  customers  in  registering 
complaints  about  IS&S  products  and  services. 


While  there  is  widespread  use  of  annual  customer  surveys,  most 
organizations  note  that  they  are  very  limited  in  the  information  they 
provide  that  can  tie  to  corrective  action  strategies.  One  manager  said  his 
organization  was  putting  customer  surveys  on  hold  until  the  organization 
could  develop  questions  which  would  provide  answers  for  concrete 
corrective  action.  Another  manager  said  he  planned  to  do  "just  in  time" 
surveys  on  specific  projects  or  IT  processes  instead  of  a  survey  at  the 
end  of  the  year.  His  rationale  was  that  year  end  surveys  only  capture 
problems  or  feelings,  while  he  wanted  to  capture  data  that  the  IT 
organization  could  take  action  on,  that  is  linked  to  organizational 
activities,  and  where  he  can  see  trends  in  particular  areas  and  focus  on 
skills  and  problem  areas.  His  point  was  that  asking  general  questions  is 
no  way  to  develop  actionability  or  determine  significant  trends. 

Managers  in  the  organizations  we  studies  emphasized  that  it  is  important 
to  have  consistency  of  some  hard  data  collection  from  year  to  year. 

Some  performance  information  such  as  computer  and  communications 
availability,  customer  satisfaction  percentages,  and  software  capability 
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maturity  levels  are  relatively  durable  and  comparable  over  longer  time 
periods.  These  measures  track  trends  in  areas  where  products  and 
services  do  not  change  significantly.  One  manager  suggested  that 
measures  should  not  be  policy-based  as  policy  can  change  from  year  to 
year. 

2.  Develop  and  Use  Baseline  and  Benchmark  Information 

The  organizations  we  studied  spent  considerable  time  and  effort  on 
baselining  and  benchmarking,  two  entirely  different  activities.  They 
assessed  what  performance  information  they  had  for  the  measures  they 
had  selected  (baselining)  and  how  that  information  might  compare  to 
that  of  other  organizations  or  similar  processes  within  their  organization 
if  there  were  discrete  IT  units  (benchmarking). 

In  baselining,  available  IT  performance  information,  or  information  that 
will  have  to  be  collected,  becomes  part  of  the  performance  baseline  for 
each  scorecard  objective.  The  current  performance  becomes  the 
"baseline"  against  which  further  performance  is  measured.  Without 
baselining,  there  is  no  standard  to  measure  progress.  In  fact,  one  of  the 
initial  tasks  in  IT  performance  management  is  determining  current 
performance  using  the  measures  designed  for  the  balanced  scorecard  or 
a  similar  approach.  The  organizations  we  studied  found  that 
performance  data  within  a  balanced  scorecard  cannot  be  a  set  of  data  on 
top  of  all  the  IT  performance  data  that  was  previously  collected. 

To  set  actual  IT  performance  targets,  organizations  often  do 
benchmarking,  an  activity  that  is  much  different  from  baselining. 
Benchmarking  was  done  with  other  IT  organizations  in  the  enterprise, 
with  IT  organizations  outside  the  enterprise,  or  with  similar  processes 
but  in  other  industries.  For  example,  Kodak  benchmarks  with  companies 
in  markets  where  it  competes  (competitive  benchmarking),  with  leading 
or  best  of  class  organizations  within  any  industry  (functional 
benchmarking),  and  among  the  operating  units  within  Kodak  (internal  ; 
benchmarking).  For  example,  handling  customer  support  phone  lines 
could  be  benchmarked  against  mail  order  telephone  operations. 

Another  organization  we  studied,  Xerox  Corporation,  was  outsourcing 
many  of  its  IT  operations  to  Electronic  Data  Systems  (EDS).  Xerox  is 
benchmarking  EDS  service  content,  service  delivery,  and  pricing  against 
the  best  in  many  countries.  Initially,  Xerox  established  a  price  and 
service  baseline  for  comparison  of  EDS  services  and  prices  against  the 
best  organizations. 

Many  of  the  organizations  we  studied  have  made  benchmarking  an 
integral  part  of  their  IT  performance  management  activities. 
Benchmarking  information  is  often  required  in  decision-making  packages 
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sent  forward  to  senior  executives.  Most  have  devoted  at  least  one  staff 
member  to  benchmarking.  Texas  Instruments,  for  example,  has  a 
corporate  office  of  benchmarking  and  best  practice  sharing. 

Some  organizations  we  studied  cautioned  that  benchmarking  requires 
focus.  Benchmarking  often  requires  looking  at  process  improvements 
versus  strategic  value.  While  an  organization  will  know  how  an 
organization  does  a  particular  IT  process  or  set  of  activities,  that 
knowledge  may  do  little  to  improve  outcomes.  If  the  benchmarking 
focus  is  on  key  measures,  such  as  reducing  cycle  time  or  how  well  IT 
business  process  support  customers,  then  benchmarking  realizes  a 
strategic  potential.  The  organizations  use  baselining  and  benchmarking 
information  to  identify  performance  gaps  between  current  IT 
performance  and  desired  achievement  levels.  Leading  organizations 
recognize  that  improvement  goals  must  flow  from  a  fact-based  analysis 
of  IT  performance  aligned  to  organization  mission.  At  least  one  of  the 
organizations  we  studied  believed  that  goals,  such  as  six  sigma,  make 
benchmarking  irrelevant.16  The  standard  for  performance  is  already  set 
at  a  "zero  defect"  goal.  One  manager  noted  that  "the  key  to  performance 
is  how  the  [IT]  organization  supports  the  business,  not  how  the  IT 
organization  compares  to  other  IT  organizations."  However,  others 
believed  that  benchmarking  provides  comparison  data  on  exemplary 
organizations  and  sets  "stretch"  performance  standards  for  the  IT 
organization. 

3.  Assess  Performance  Maturity  and  Develop  Complete 
Performance  Definitions 

A  common  stumbling  block  for  organizations  is  the  tendency  to  struggle 
to  develop  "perfect"  measures  instead  of  thinking  in  terms  of  improving 
measures  over  time.  But  as  one  manager  told  us,  in  performance 
measurement,  you  simply  cannot  be  good  at  everything  right  away  which 
forces  you  to  undertake  a  phased  approach.  As  an  organization  moves 
gains  more  performance  management  experience,  better  and  more 
appropriate  goals  will  be  defined  so  the  supporting  measures  will,  in 
turn,  be  modified.  For  many  measures,  the  definitions,  data  collection 
techniques,  and  reporting  will  need  to  be  refined  over  time. 

Measure  maturity  assessment  can  be  a  part  of  the  measure  definitions 
that  organizations  develop.  Generally,  these  definitions  cover  what  the 
measure  is  intended  to  show  and  why  it  is  important,  how  performance 
data  are  generated,  who  is  responsible  for  collecting  it,  how  the  measure 


16Six  sigma  is  a  measure  defining  the  quality  level  of  a  product,  service,  or  process.  This 
organization  defined  six  sigma  is  99.9997  percent  perfect  or  3.4  defects  per  million 
opportunities  to  create  a  defect.  If  set  as  a  goal,  it  basically  calls  for  a  virtual  "zero  defect" 
standard. 


GAO/AIMD-98-89  IT  Performance  Measurement  Guide 


Page  59 


is  specifically  calculated,  any  limitations  placed  on  the  measurement  data 
(for  example,  factors  beyond  the  organization's  control),  and  whether  the 
data  is  cumulative  or  noncumulative.  As  this  process  is  repeated  and 
refined  over  time,  there  is  increased  confidence  that  the  measure 
accurately  describes  performance  in  the  relevant  area. 


Case  Study  8: 

Gaining  Experience  with  Fundamental  Measures  and  then  Expanding  Out 

Kodak  is  one  organization  that  is  systematically  defining  the  maturity  of  each  measure  it 
plans  to  use  in  its  balanced  scorecard.  Kodak  categorizes  measure  maturity  as  either 
fundamental,  growing,  or  maturing.  Established  indicators  are  considered  as  fundamental. 
Growing  measures  are  evolving  from  the  fundamental,  but  are  not  the  best  they  can  be. 
Maturing  measures  are  defined  as  best-in-class  for  whatever  they  are  measuring.  For  example, 
for  internal  performance  a  fundamental  measure  is  to  meet  all  service-level  agreements,  a 
growing  measure  is  information  delivery  excellence,  and  a  maturing  measure  is  defect-free 
products  and  services.  Kodak  believes  it  is  important  to  build  the  right  fundamental  practices 
first  in  developing  an  initial  IT  performance  management  system. 

As  part  of  its  development  of  measures  for  EDS  services,  Xerox  is  starting  with  what  it  calls 
"primitive  metrics. "  These  include  size,  measured  in  function  points ;  effort,  measured  in  work 
hours;  defects,  measured  by  number  of  defects ;  changes,  measured  by  number  of  changes;  and 
duration,  measured  by  elapsed  days.  Over  time,  quarterly  performance  reviews  and 
examination  of  measures  are  expected  to  result  in  revisions  to  these  measures. 


4.  Utilize  Concise,  Understandable  Performance  Reporting 

Leading  organizations  take  great  care  in  designing  IT  performance 
reports  that  are  concise,  easy  to  understand,  and  tailored  to  various 
management  needs  and  audiences.  Executive  managers,  in  particular, 
often  require  data  presentations  and  displays  that  focus  on  bottom  line 
performance  results.  When  presented  in  this  manner,  they  can  quickly 
digest  information,  focus  on  problem  areas,  seek  pertinent  follow-up 
data,  and  be  more  efficient  in  making  or  recommending  project  or 
program  decisions. 

Performance  reports  should  be  tailored  for  the  audience  for  which  the 
information  is  intended.  Operational  managers  may  need  more  details 
and  supporting,  contextual  information  while  external  stakeholders  may 
require  far  less. 
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Case  Study  9: 

Presenting  Performance  Data  in  Usable  Formats 

The  Immigration  and  Naturalization  Service's  Office  of  Information  Resources  Management 
(OIRM)  collects  information  on  both  its  Information  Technology  Partnership  contract  and  on 
program-level  performance  regarding  mission  outcomes.  The  contract  is  run  by  INS  in 
partnership  with  Electronic  Data  Systems  (EDS). 

The  following  figure  shows  how  overall  contract-level  performance  is  reported  on  a  quarterly 
basis  for  each  of  the  four  key  critical  success  factors:  (1 )  customer  satisfaction,  (2)  technical 
performance,  (3)on  time,  and  (4)  within  budget.  These  quarterly  measures  can  then  be  plotted 
to  show  a  performance  trend  over  the  life  of  the  contract.  Gathering  data  such  as  this  will  also 
allow  INS  to  provide  documented  and  objective  feedback  on  a  contractor’s  past  performance. 


Case  Study  9: 

INS/ITP  Quarterly  Performance  Reporting 


Percent 


1st  Qtr.  2nd  Qtr.  3rd  Qtr.  4th  Qtr. 

■Oust.  Sat.  ■  Budget 
■On  Time  HiTech.  Pert. 


1st  Qtr. 
Performance 
Measures 

Customer 

Satisfaction 


Within  Budget 


Score  Weight  Weighted 
Score 


On  Time 


Technical 

Performance 


ITP  Performance  Index  79.8 
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Case  Study  10: 

Effective  Presentation  of  Performance  Data  and  Contextual  Information 


The  Minnesota  Department  of  Administration  provides  business  management  and 
administrative  services  to  Minnesota  agencies .  Most  of  the  Department's  operations  are  fee- 
based  operations  in  areas  such  as  data  processing ,  printing ,  vehicle  rental ,  and  the  sale  of 
office  supplies .  The  Department's  InterTechnologies  Group  (InterTech)  provides  services  in 
managing  and  operating  information  technology  resources. 

Figure  21  illustrates  one  objective  for  customer  service  performance.  The  objective  is  written 
with  a  specific  performance  rating  standard  and  the  use  of  a  specific  survey  for  data 
collection.  The  performance  data  are  reported  in  this  format ,  with  the  objective ,  definition , 
rationale ,  and  data  source.  The  data  also  include  a  discussion  of  past  performance  and  the 
plan  to  achieve  targets.  These  data  and  reports  on  other  measures  are  given  to  the  Minnesota 
Legislature  in  a  formal  performance  report. 


Case  Study  10: 

Minnesota  Department  of  Administration’s  Customer  Service 
Performance  Data  Template 


Objective  1:  InterTech  will  receive 
above-average  customer  service  rating  (i.e.  a 
rating  of  3.0  or  better  on  a  scale  of  1 .0-4.0)  for 
key  services,  as  rated  by  customers  on  an 
annua!  customer  service  survey. 

DEFINITION,  RATIONALE,  DATA  SOURCES 

Definition:  Key  services  are  defined  as  those  for 
which  customers'  ratings  averaged  3.0  or  greater, 
indicating  important  or  very  important.  In  the  FY  1 
survey,  27  services  were  measured  with 
responses  from  259  customers.  Customers  rated 
15  of  21  data  processing  services  as  important 
and  8  of  11  telecommunications  services  as 
important.  The  reported  satisfaction  ratings  are 
based  on  these  services. 

Rationale:  Above-average  satisfaction  is  defined 
as  customer  ratings  averaging  3.0  or  greater, 
indicating  satisfied  or  very  satisfied.  The  future 
year  goal  is  to  improve  any  services  rated  below 
3.0  by  at  least  10%  each  year  and  to  improve  any 
services  rated  3.0  or  above  by  at  least  5%  each 
year,  until  a  perfect  score  of  4.0  is  achieved. 

Data  sources:  Annual  customer  satisfaction 
survey. 


Actual  performance 

Fiscal 
Year  1 

Fiscal 
Year  2 

Telecom.  Services 

Perform.  Targets 

3.0 

3.3 

Actual.  Perform. 

3.1 

Data  Processing 

Services 

Perform.  Target 

3.0 

3.4 

Actual  Perform. 

3.2 

DISCUSSION  OF  PAST  PERFORMANCE: 

Prior  to  FY  1,  InterTech  periodically  surveyed 
customer  agencies  to  determine  their  level  of 
satisfaction  with  key  services.  This  effort  was 
expanded;  surveys  are  now  conducted  annually. 


PLAN  TO  ACHIEVE  TARGETS: 

Targets  in  future  years  will  be  based  on  the 
previous  year's  percent  improvement. 
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5.  Conduct  Measure  Reviews  and  Audits 

Most  organizations  we  studied  regularly  assess  their  measures  to  see  if 
they  are  still  appropriate  for  measuring  results  and  assigning 
accountability.  Most  of  these  organizations  do  regular  external  audits  of 
IT  measure  appropriateness  and  data  collection  efficiency.  One  manager 
said,  "We  believe  any  successful  organization  needs  a  good  basic 
independent  look  at  measures,  challenging  what  are  the  right  questions." 
The  reviews  and  audits  serve  as  an  oversight  of  measures  and  the  data, 
challenging  the  necessity  of  the  measures  (i.e.,  are  they  being  used?)  and 
their  linkage  to  enterprise  strategic  plans  and  individual  performance 
expectations  (i.e.,  are  they  having  an  impact?).  Reviewers  consider  the 
accuracy,  completeness,  timing,  match  to  actual  conditions,  and  technical 
features  such  as  the  method  of  results  calculations. 

For  example,  as  new  technology  is  introduced,  measures  can  become 
obsolete  or  less  important.  Or  customer  measures  supplant  or  replace 
traditional  measures.  In  one  organization,  IT  used  thousands  of  lines  of 
code  per  calendar  month  as  a  measure  of  software  productivity.  From 
the  customer  point  of  view,  the  measure  was  ineffective,  and  the 
measure  was  changed  to  cycle  time.  Over  time  the  organization  will 
develop  more  reliable  measures. 

The  reviews  also  analyze  key  performance  drivers  in  each  goal  area. 

They  question  what  a  stress  in  one  area  will  do  to  the  results  in  other 
areas.  They  also  can  question  what  performance  gain  might  be  achieved 
for  a  certain  level  of  performance  effort,  and  if  the  performance  results 
are  worth  that  effort. 


How  to  Get  Started 

To  build  a  comprehensive  measure,  data  collection,  and  analysis 

capability,  organizations  should 

•  designate  specific  IT  staff  to  gain  skills  in  measurement,  data 
collection,  and  analysis, 

•  review  existing  data  collection  and  reporting  and  determine 
what  is  still  appropriate  and  what  should  be  changed  or  deleted 
to  match  possible  measurement  scorecard  areas,  and 

•  determine  what  are  preliminary  interdependencies  among 
scorecard  goal  areas,  objectives,  and  measures. 
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Practice  Area  5: 
Strengthen 
IT  Processes  to 
Improve  Mission 
Performance 


ALIGN 


REINFORCE 


Practice  Area  Characteristics 

1.  Define  the  IT  business  processes  that  produce  IT  products 
and  services  meeting  mission  goals 

2.  Using  IT  performance  information,  prioritize  IT  business 
processes  essential  for  improving  mission  performance. 


Practice  Area  Overview 

In  many  of  the  organizations  we  studied,  business  process  improvement 
is  a  high  priority  enterprise  strategy.  Often,  enterprise  and  operational 
customer  business  process  improvement  can  only  be  accomplished  using 
IT  products  and  services.  That  means  that  IT  must  ensure  that  its  own 
business  processes  are  the  best  they  can  be.  Simply  put,  IT  internal 
business  processes  deliver  IT  products  and  services  which  the  enterprise 
and  operational  customers  depend  on  to  support  their  missions.  If  IT 
does  not  have  the  capability  to  deliver  high  quality  products  and  services, 
then  organizational  goals  can  suffer.  This  is  an  important  reason  why 
the  balanced  scorecard  approach  includes  internal  IT  business  processes. 

Practice  Area  Characteristics 

1.  Define  IT  Business  Processes  That  Produce  Products 
and  Services  Critical  for  Meeting  Mission  Goals 

All  leading  organizations  define  their  key  IT  business  processes.  This 
helps  the  IT  organization  focus  on  primary  activities,  identify  IT 
competencies,  eliminate  processes  which  do  not  add  value,  and  facilitate 
IT  process  innovation.  Several  of  the  organizations  we  studied  noted 
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that  process  measures  are  tightly  linked  with  tasks  or  activities.  Should 
organizational  structures  change  and  tasks  and  activities  move  within  the 
IT  organization,  measures  can  more  easily  move.  In  other  words,  IT 
processes  and  subprocesses,  once  defined,  are  "portable"  from  one  part 
of  the  organizational  structure  to  another. 

Some  of  the  organizations  developed  their  IT  process  orientation  based 
on  work  done  by  the  Ernst  and  Young  Center  for  Information 
Technology  and  Strategy  and  later  published  by  a  workgroup  of  the 
Society  for  Information  Management  (SIM).17  SIM's  Information  Systems 
Process  Architecture  (ISPA)  process  framework  is  a  model  of  how  a 
typical  organization  obtains  and  applies  information  systems  and 
technology.  In  describing  its  initial  framework  issued  in  September  1993, 
SIM  indicated  that  its  process  model  (1)  defines  strategically  important 
IT  processes  in  an  overall  framework,  (2)  communicates  with  IT 
stakeholders  on  the  value,  activities,  and  organization  of  IT,  and  (3) 
provides  a  basis  for  allocating  IT  resources  that  is  congruent  with 
activity-based  costing. 

As  shown  in  figure  13,  SIM's  ISPA  version  2.0  process  framework,  issued 
in  March  1996,  includes  eight  IT  processes  which  overlap.  This 
framework,  like  the  earlier  version,  provides  an  example  of  a  way  to 
organize  IT,  determine  core  competencies,  and  identify  process  owners. 

Figure  13:  ISPA  Version  2.0  Process  Framework 


17 

See  the  Society  for  Information  Management  Working  Group  on  ISPA  process  architecture 
documents  listed  in  the  bibliography. 
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As  explained  below,  each  process  in  this  suggested  framework  has  a 
specific  purpose  and  suggested  metrics  (measures). 

•  Perform  Customer  Relations 

This  process  is  used  to  develop  and  maintain  working  relationships  with 
the  customers  of  IT  products  and  services.  Performance  metrics  focus 
on  the  business  impact  of  IT,  customer  satisfaction,  and  product  and 
service  measurements,  such  as  the  accuracy  and  response  time  for 
solving  problems  or  number  of  viable  IT  projects  identified. 

•  Market  IT 

This  process  is  used  to  ensure  that  customers  want,  need,  and  buy  the 
products  and  services  offered.  Sample  metrics  focus  on  the  increased 
business  value  of  IT,  IT  return  on  investment,  customer  satisfaction,  and 
product  and  service  effectiveness  measures. 

•  Align  Business  and  IT 

This  process  is  used  to  incorporate  IT  into  strategic  business  change 
activities  in  a  way  that  captures  opportunities  from  current  and  emerging 
technologies,  and  to  promote  process  innovation  leadership  using  IT  as 
the  catalyst  and  using  proven  visioning  and  change  management 
techniques.  Sample  metrics  focus  on  the  business  value  of  IT,  net 
present  value  of  projects  approved  by  the  strategic  business  unit,  and 
improvements  to  business  processes. 

•  Manage  Enterprise  Architecture 

This  process  is  used  to  provide  a  framework  for  delivering  consistent 
products  and  services.  Sample  metrics  focus  on  IT  architecture  design 
and  implementation,  the  number  of  viable  IT  projects  identified,  how 
service  chargebacks  align  with  customer  views  of  services,  and  the 
degree  of  technology  standardization. 

•  Develop  and  Deploy  Products  and  Services 

This  process  is  used  to  acquire,  develop,  deliver,  and  implement  new 
information  services  for  the  organization.  Sample  metrics  might  include 
the  percent  of  projects  on-time  and  with  the  desired  functionality, 
projects  completed  within  budget,  customer  satisfaction,  and  degree  of 
technology  usage  in  conducting  core  business  processes. 

•  Deliver  and  Support  the  Products  and  Services 

This  process  is  used  to  ensure  that  the  products  and  services  are 
deployed  in  the  most  effective  and  efficient  manner.  Metrics  can  focus 
on  customer  satisfaction  survey  ratings,  increased  demand  for  IT 
information,  adequacy  of  equipment  and  facilities,  availability  and 
accessibility  of  data,  number  of  problems  received  and  requests  satisfied, 
and  mean  time  to  data  recovery. 
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•  Plan  the  IS  Organization 

This  process  is  used  to  shape  and  support  business  unit  strategies, 
establish  strategy  and  vision  for  long-term  IS  use,  develop  tactical  plans 
and  development  and  infrastructure  resources  over  a  12  to  18  month 
horizon,  and  design  key  processes  within  the  IS  organization.  Sample 
metrics  could  include  employee  satisfaction  with  the  IS  vision  and 
knowledge  about  technology  plans. 

•  Manage  IS  Organization  Business 

This  process  is  used  to  manage  the  processes  within  IS  that  deal  with  the 
health  and  state  of  the  IS  organization,  its  employees,  and  vendors. 
Sample  metrics  include  an  employee  satisfaction  and  commitment  index, 
team  efficiency  and  effectiveness  measures,  product  and  service 
measurements,  and  an  index  of  employee  skills. 


Case  Study  11: 

Texas  Instruments  IT  Process  Segmentation  Effectively  Supports  Business 
Needs 

At  Texas  Instruments,  many  IT  customer  business  process  reengineering  activities 
established  a  clear  need  for  rapid  provisioning  of  IT  solutions,  flexibility  with  respect  to 
business  rules  and  work  flows,  and  cost  effective  use  of  advanced  technology.  In  response, 
the  Information  Systems  and  Services  Group  (IS&S)  developed  its  IT  process  map  with 
five  processes,  as  shown  in  the  following  figure. 

The  process  map  defines  activities  that  "map"  into  both  the  operational  customer  and  IT 
organization  sphere.  For  example,  strategy  management  as  a  process  is  the  joint 
responsibility  of  both  the  business  customers  and  IS&S  (the  enterprise  IT  group). 

Strategy  management  sets  the  vision  and  timing  of  all  the  elements  of  IT  needed  to 
support  Texas  Instruments.  Solution  provisioning  is  the  mechanism  for  assembling  the 
hardware  and  software  pieces  together  to  form  solutions  for  IT  customers.  The  service 
and  product  support  area  deploys  and  maintains  IT  products  and  services.  The  research 
and  architecture  definition  area  does  research,  experiments  with  and  sets  standards  and 
methodologies  for  current  and  future  architectures.  The  component  provisioning  area 
provides  reusable  applications,  building  blocks,  assembly  tools,  and  integration  expertise. 
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2.  Using  IT  Performance  Information,  Prioritize  IT  Business 
Processes  Essential  for  Improving  Mission  Performance 

Given  the  many  IT  business  processes  that  an  IT  organization  manages, 
which  ones  are  the  most  important  for  improvement?  In  the 
organizations  we  studied,  customer  business  process  improvement 
strategies  and  performance  requirements  frequently  identify  and  drive 
major  improvements  in  IT  business  process.  They  key  is  to  determine 
the  most  efficient  and  effective  manner  to  organize  roles,  responsibilities, 
and  delivery  of  IT  products  and  services  within  the  organization.  This 
requires  a  clear  understanding  of  what  IT  functions  are  the  domain  of 
business  units,  unit-organized  IT  groups,  and  the  corporate  or 
enterprisewide  IT  organization.  Knowing  how  effective  is  this 
arrangement  is  in  providing  effective  IT  support,  which  IT  functions  are 
performing  well,  and  where  service  improvements  are  needed  is  critical 
to  targeting  IT  management  attention. 
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Case  Study  12: 

Using  Business  Process  Focus  to  Shape  IT  Service  Delivery 

Kodak  is  deciding  what  IT  processes  it  should  pursue  to  support  the  enterprise  and  operational 
customers'  business  processes.  The  focus  is  to  look  at  each  business  process  and  its  use  of  IT  and 
determine  if  the  information  systems  organization  identified  the  right  technology  for  the  business 
needs . 

For  Kodak ,  enterprise  goals  drive  a  process  focus  for  strategic  business  units  and  support  functions 
such  as  IT.  The  company's  enterprise  goals  are  three  fold: 

(1)  a  10  time  defect  reduction  every  3  years , 

(2)  reaching  6  sigma  by  the  year  2000 ,  and 

(3)  a  10  time  cycle  time  reduction  in  3  years. 

Kodak  defines  a  defect  as  a  variation  in  a  product  or  service  which ,  if  not  caught ,  prevents  the 
company  from  meeting  the  needs  of  its  customers.  Cycle  time  is  the  total  time  to  move  a  unit  of 
work  from  the  beginning  to  the  end  of  a  process.  Kodak  believes  that  maximizing  business  processes 
is  the  only  way  to  achieve  these  dramatic  improvement  goals. 


How  to  Get  Started 

To  begin  focusing  resources  on  IT  process  improvement, 
organizations  should 

•  use  a  model  such  as  that  suggested  by  the  Society  for 
Information  Management  to  define  IT  business  processes, 

•  determine  which  IT  business  processes  are  the  most  critical 
to  meeting  enterprise  and  operational  customer 
performance  goals  and  requirements,  and 

•  choose  one  or  more  critical  IT  business  processes  to 
baseline  existing  performance  and  benchmark  against 
industry  standards  or  leading  public/private  organizations,  : 

_ _ _ 
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Key  Lessons  Learned  for  Effective 
Implementation 


Agency  managers  just  starting  to  develop  IT  performance  management 
systems  or  those  who  want  to  enhance  existing  ones  have  a  formidable 
task.  They  often  are  faced  with  resource  constraints,  demands  for 
immediate  IT  support  and  solutions  as  program  areas  reduce  staff  and 
reengineer  their  business  processes,  and  skepticism  about  the  value  of 
performance  management. 

From  the  organizations  we  studied  and  the  experiences  of  other 
organizations  described  in  the  literature,  three  key  activities  are  essential 
in  putting  the  practices  in  place.  These  are  assessing  organizational 
readiness  for  a  successful  IT  performance  management  system  and 
staging  the  system  development,  following  a  simple  measure  selection 
process,  and  recognizing  system  maturity  will  change  over  time. 

Assess  Organizational  Readiness 

The  leading  organizations  find  that  assessing  organizational  readiness  for 
a  successful  IT  performance  management  system  is  essential.  Here,  the 
organizations  look  for  the  involvement,  commitment,  and  day-to-day 
support  of  enterprise  senior  managers.  They  also  determine  if  they  have 
adequate  resources,  including  staff  allocation,  skills,  time,  tools,  and  use 
of  consultants  or  technical  assistance  if  needed.  A  manager 
characterized  a  good  performance  management  system  as  one  that  "has 
complete  buy-in  from  top  management,  involves  front  line  employees  in 
system  design,  and  results  in  front  line  employees  understanding  what 
they  do  and  why  they  are  doing  it." 

Organizational  readiness  also  means  making  sure  that  existing  planning 
and  decision  making  structures  can  accept  performance  results  so  they 
can  be  used.  As  the  introductory  chapter  to  this  guide  explained, 
performance  measures  are  a  central  piece  of  alignment  around  mission 
planning,  budgeting,  and  evaluation.  These  are  essentially  separate 
processes  linked  by  performance  measures.  The  organization  needs  the 
capability  to  specify  clear  goals  and  objectives  to  set  the  focus  and 
direction  of  IT  performance,  creating  an  IT  performance  improvement 
plan  and  revisiting  it  every  one  or  two  years.  The  IT  organization  has  to 
understand  the  business  of  operational  customers  and  make  sure  IT 
measures  are  consistent  with  business  measures.  That  means  the 
capability  to  develop  a  "theory"  of  how  IT  supports  enterprise  and 
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operational  customers  so  the  organization  can  build  the  chain  of  IT 
events  and  evidence  described  in  practice  1. 

The  organizations  also  determine  if  they  have  the  support  of  other 
stakeholders  and  funding  sources,  such  as  legislative  staff.  As  mentioned 
under  practice  1,  stakeholders  are  one  of  the  parties  which  have  to  reach 
a  common  understanding  of  IT  goals,  objectives,  appropriate  measures, 
and  anticipated  outcomes. 

Lastly,  organizational  readiness  means  paying  attention  to  organizational 
culture  -  is  it  receptive  to  data  collection,  measurement,  and  analysis 
and  accountability  for  performance  and  decisions  as  part  of  an  overall 
performance  improvement  system?  The  organization  should  have  a 
philosophy  that  is  positive  towards  performance  management  and 
measurement  and  views  it  as  a  way  to  focus  on  quality  and  operational 
customer  satisfaction.  That  means  the  organization  should  be  willing  to 
assess  organizational  values,  principles,  and  how  they  are  working.  That 
is  the  key  to  success,  one  manager  explained,  "We  have  the  culture  to 
support  this  -  people  get  to  where  they  track  and  report  metrics,  they 
think  about  quality  goals.  [IT  performance  management]  is  a  hard  sell 
where  there  is  not  a  total  quality  culture,  where  the  focus  is  on 
technology,  not  satisfying  the  customer." 

Follow  a  Simple  Measure 
Selection  Process 

Selecting  and  implementing  IT  performance  measures  is  extremely 
complex.  Each  enterprise  and  its  operational  customers  have  a  mission 
and  goals  that  often  differ  significantly  from  other  agencies.  A  set  of  IT 
performance  measures  that  work  for  one  organization  likely  will  not 
completely  work  for  another  organization.  The  performance  measures 
differ  on  what  is  valued  most  in  terms  of  IT  performance.  To  illustrate, 
IT  goals  and  objectives  which  stress  cost  savings  would  have  more 
measures  related  to  efficiency  versus  effectiveness.  IT  goals  and 
objectives  stressing  customer  satisfaction  and  other  service-oriented 
goals  might  have  fewer  efficiency  measures.  However,  well-managed  IT 
activities  should  take  a  balanced  approach  to  goals,  objectives,  and 
related  measures. 

In  the  organizations  we  studied  and  in  the  literature  we  reviewed,  one 
element  of  success  is  sifting  through  the  many  possible  goals,  objectives, 
and  measures  before  finalizing  them  in  a  balanced  scorecard  or  similar 
approach.  The  sifting  process  identifies  potential  objectives  and 
measures  for  all  parts  of  IT  activities.  It  assesses  which  measures  will 
be  valuable  for  which  purposes  and  to  whom  and  eliminates  measures 
which  are  not  relevant  to  customer  and  stakeholder  needs.  And  it 
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eliminates  measures  for  which  good  quality  data  cannot  be  practically 
obtained.  One  manager  said,  "We  want  to  put  the  best  measures  in 
place.  We  want  them  simple,  collectable,  repeatable,  and  concrete." 


A  good  example  of  the  selection  process  is  the  measurement  "roadmap" 
followed  by  the  Minnesota  Department  of  Transportation.  The  roadmap 
examines  and  reduces  the  total  potential  to  a  vital  few  number  of  goals, 
objectives,  and  related  measures.  The  roadmap,  shown  in  figure  14  and 
enhanced  with  additional  information  from  the  United  Kingdom's  Royal 
Mail,  is  a  general  performance  measure  selection  framework  that  all 
program  and  support  functions  follow.  In  the  roadmap,  the  word 
"organization"  can  refer  to  an  agency,  a  program,  or  a  support  function. 


Figure  14:  IT  Measurement  Implementation  Roadmap 


Stage  I:  Clarify  Goals  and  Objectives 


Source:  Minnesota  Dept,  of  Transportation  and  United  Kingdom's  Royal  Mail 


In  stage  1,  the  organization  clarifies  its  overall  goals  and  objectives,  first 
by  fulling  listing  all  possibilities,  and  then  converging  on  the  vital  few 
goals  and  objectives.  In  stage  II,  the  organization  would  develop  its 
measures  and  make  sure  its  measurement  system  has  a  diversity  of 
measures  and  follows  a  tiering  approach.  In  the  last  stage,  the 
organization  uses  the  measures,  establishing  baselines  and  targets, 
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comparing  against  benchmarks,  and  monitoring  progress  for  continual 
improvement. 

For  support  functions  such  as  IT,  two  roadmaps  are  in  play.  One 
roadmap  develops  the  objectives  and  measures  for  an  agency  or  program 
area;  another  roadmap  takes  that  agency  or  program  information  and 
develops  objectives  and  measures  for  IT.  This  is  the  intent  of  the 
balanced  scorecard  approach  discussed  in  practice  2. 

Develop  IT  Performance  Management 
System  Maturity  Over  Time 

In  discussing  the  development  of  an  IT  performance  management  system, 
we  attempted  to  determine  if  there  was  a  maturity  aspect  to  system 
development.  Several  of  the  managers  we  talked  to  believe  there  is  a 
staging  in  performance  management  emphasis  and  expectations  although 
staging  is  not  precise  or  formal.  Figure  15  shows  possible  stages  in 
performance  management  and  their  linkage  to  the  balanced  scorecard 
areas,  drawing  on  work  done  for  the  city  of  Phoenix  and  discussed  in  the 
literature.  While  shown  as  discrete  stages  for  purposes  of  illustration, 
the  distinction  between  them  often  is  not  clear-cut  as  they  overlap  over 
time. 
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Figure  15:  Stages  in  IT  Performance  Management 


1.  Traditional 
Internal  IT 
Operations 
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Source:  Howard  Rubin,  Ernst  &  Young,  City  of  Phoenix 
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The  premise  is  that  IT  organizations  have  to  be  good  at  the  basics  in 
stage  one  -  traditional  internal  IT  operations  -  and  then  move  to  stages 
two  (linking  enterprise  and  customer  mission  with  new  and  improved  IT 
services)  and  three  (be  mission-results  oriented).  In  other  words,  an  IT 
organization  that  is  viewed  as  a  failure  in  day-to-day  operations  will  not 
have  the  credibility  nor  support  of  the  rest  of  the  organization  to  play  a 
strategic  role  directly  tied  to  mission  results. 

In  stage  one,  the  IT  organization  is  developing  and  implementing  a 
performance  management  system  that  examines  internal  operations 
against  standards  and  acceptable  levels  of  performance.  Traditional, 
activity-based  measures  such  as  number  of  reports  issued  or  mainframe 
availability  are  used.  In  stage  two,  the  IT  organization  is  eliminating  root 
causes  of  defects,  building  its  competencies  to  consistently  deliver  new 
and  improved  systems  and  solutions  on  time  and  within  budget,  and 
linking  operational  measures  to  mission  performance.  The  goal  is  to 
improve  IT  processes  and  prevent  defects.  For  both  stage  one  and  two, 
most  of  the  measures  fall  in  the  internal  business  goal  and  innovation 
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and  learning  goal  of  the  balanced  scorecard  approach  discussed  in 
practice  2. 

In  stage  three,  the  IT  organization  has  the  capability  of  applying  IT 
expertise  to  enterprise  and  operational  customer  mission  requirements 
and  putting  its  IT  outcomes  in  those  mission  terms.  Measures  are  based 
on  customer  needs  and  benefits,  expressing  measures  in  terms  customers 
understand,  such  as  business  outcomes.  Stage  three  IT  organizations 
also  specify  who  is  responsible  for  corrective  action.  In  this  stage,  the 
measures  found  in  the  strategic  and  operational  customer  goals  of  the 
balanced  scorecard  approach  are  prominent. 

The  staging  or  maturity  perspective  suggests  that  organization  should 
consider  using  the  balanced  scorecard  approach  in  a  building  block 
approach.  The  organization  might  initially  develop  major  goals, 
objectives,  and  measures  in  each  of  the  four  areas.  However,  it  is 
possible  that  it  likely  must  perform  well  in  stages  one  and  two,  reflecting 
the  balanced  scorecard  areas  of  internal  business  and  innovation  and 
learning,  before  it  can  perform  well  in  strategic  and  operational  customer 
areas. 

A  Final  Note 

The  organizations  we  studied  cautioned  that  the  practice  of  IT 
performance  management  and  measurement  is  in  its  infancy.  Most  of 
the  organizations  we  studied  have  worked  on  their  IT  performance 
management  systems  for  several  years  and  most  of  those  efforts  are  part 
of  strong  performance  goals  at  the  enterprise  level.  The  cities  of 
Phoenix  and  Sunnyvale,  for  example,  have  long-standing  reputations  for 
being  well-managed  and  their  general  and  specific  IT  measurement 
approaches  have  evolved  over  many  years. 

In  both  these  organizations,  as  with  others  we  studied,  there  is  a  strong 
performance  management  culture.  The  organizations  share  many  similar 
performance  management  values  and  management  objectives  that  stress 
IT  results  and  accountability.  For  them,  IT  measures  make  a  valued  and 
positive  difference  in  mission  and  business  performance. 
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Objectives,  Scope,  and  Methodology 

The  objectives  of  our  research  were  to  (1)  identify  information  technology  performance 
management  practices  used  by  leading  private  and  public  sector  organizations  with  demonstrated 
success  in  developing  and  using  information  technology  solutions  and  infrastructure  to  improve 
mission  or  business  performance  and  outcomes,  and  (2)  share  our  results  with  federal  agencies  to 
help  improve  overall  mission  performance. 

Scope 

Our  research  focused  on  information  technology  performance  management  practices  used  by 
management  teams  and  staff  in  five  private  sector  companies,  three  state  and  local  governments, 
and  three  federal  agencies.  The  organizations  were  chosen  purposefully,  not  at  random  or  to 
ensure  representation  of  a  larger  group.  We  selected  the  private,  state,  and  local  organizations 
based  on  (1)  recognition  by  other  organizations  and  independent  researchers  for  their  progress  in 
successfully  developing  and  using  information  technology  performance  management  systems,  (2) 
recognition  by  professional  publications  for  their  performance  management  systems,  and  (3) 
willingness  to  participate  as  a  case  study  organization.  The  federal  agencies  were  selected  based 
on  their  recognition  by  federal  information  resources  management  officials  for  initiating 
comprehensive  work  on  information  technology  performance  management.  Because  our  work 
often  involved  data  that  these  organizations  regarded  as  proprietary  or  sensitive,  we  agreed  not  to 
disclose  any  data  they  wished  to  protect. 

To  supplement  our  findings  from  these  private  and  public  sector  organizations,  we  gathered 
additional  information  from  other  federal,  state,  and  local  organizations.  This  information 
included  both  generic  and  information  technology  performance  management  information,  ranging 
from  guides  to  specific  practices. 

Methodology 

Our  research  was  conducted  with  an  illustrative  case  study  approach  using  open-ended  and 
focused  interviews  and  documentary  analysis,  not  direct  observations.  In  conducting  the  case 
studies,  we  interviewed  senior  executives,  line  managers,  and  information  technology 
professionals  to  learn  how  the  organization  measured  and  managed  the  contribution  of 
information  technology  towards  organizational  goals.  Interview  information  was  supplemented  by 
documentary  analysis  of  each  organization's  information  technology  performance  management 
approach. 

For  quality  assurance,  we  conducted  a  meeting  of  case  study  participants  to  obtain  group 
comments  on  an  initial  draft  of  this  guide,  followed  by  individual  case  study  participant  reviews 
of  a  subsequent  draft.  We  also  distributed  the  draft  to  other  experts  on  information  technology 
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performance  management,  the  Office  of  Management  and  Budget,  and  the  General  Services 
Administration.  We  also  made  numerous  presentations  as  the  guide  was  developed  to  test  our 
preliminary  findings  and  the  applicability  to  the  federal  government  with  Federal  executives, 
managers,  and  staff,  representing  both  line  and  information  technology  functions.  We  have 
incorporated  changes  where  appropriate. 

Caveats 

Information  technology  performance  management  and  measurement  is  very  much  in  its  infancy  in 
both  public  and  private  sectors.  As  an  initial  step,  this  guide  presents  a  framework  that  begins  to 
document  the  state  of  the  practice  drawn  from  our  analysis  of  a  relatively  small  number  of  case 
studies.  Much  more  research  and  analysis  remains  to  be  done.  The  practices  we  have  presented 
can  serve  as  starting  point  for  any  organization,  tailored  to  the  strategic  directions  and 
performance  requirements  unique  to  each  organization. 
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Case  Study  Organizations  and  Participants 

We  would  like  to  acknowledge  the  invaluable  assistance  of  the  following  individuals  who 
serviced  as  key  contacts  for  the  case  study  organizations. 


American  Express  Travel  Related 
Services  Company,  Inc. 

Cliff  Shoung  (Project  Leader, 
Benchmarking) 

Paula  Bouthillier  (Project  Leader,  Strategic 
Advantage) 

U.S.  General  Services  Administration 

Jon  Desenberg  (Management  Analyst) 

Immigration  and  Naturalization  Service 

Janet  Keys  (Director,  Systems  Policy  and 
Planning) 

J.  T.  Lazo  (Senior  Systems  Manager,  EDS, 
INS  ITP  Contract) 

Eastman  Kodak  Company 

Jeff  Duell  (Technology  Director,  IS 
Measurement  and  Benchmarking) 

Motorola  Semiconductor  Products 
Sector 

Dorothy  Hines  (Controller,  Sector  Quality 
and  Support  Operations) 

Bret  Wingert  (Manager,  SEI  Projects) 

Oregon  Department  of  Transportation 

Craig  Holt  (formerly  Chief  Information 
Officer) 


City  of  Phoenix 

Joe  Motto  (Information  Technology 
Director) 

Bob  Wingenroth  (Deputy  City  Auditor) 

City  of  Sunnyvale 

Shawn  Hernandez  (Director,  Information 
Technology) 

Marilyn  Crane  (Information  Technology 
Administrative  Services  Manager) 

Texas  Instruments 

Gary  Pollard  (Total  Quality  Director) 
Vicki  Flewelling  (Total  Quality 
Management) 

U.S.  Department  of  Agriculture 

Joseph  Ware  (Director,  IT  Planning  and 
Program  Management) 

Xerox  Corporation 

M.  Catherine  Lewis  (Manager,  IM2000 
Integration  Planning) 

Margie  Tomczak  (Manager,  Global 
Applications) 
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Dr.  David  L.  McClure,  Senior  Assistant  Director 

Bernard  R.  Anderson,  Senior  Analyst 
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Dr.  Sharon  Caudle,  Project  Director 
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